Source-linked AI summary
GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning
Jun Chen, Yongchao Liu, Pengyu Qiu, Jiajun Zheng, Juelu Zhang, Yujie Zeng, Qin Zhang, Ziyue Qiao, Xiao Luo
TL;DR
Multi-turn RAG needs models to retrieve complete evidence chains, yet final-answer RL rewards provide sparse supervision for whether retrieval is grounded. GTA-RAG samples and validates graph-based trajectories, trains with trajectory-guided GRPO and answer-reward adaptation, and consistently outperforms RL-based RAG baselines while improving evidence-chain coverage.
Problem
Multi-hop RAG requires evidence chains across documents, but answer-level RL rewards may overlook whether the required evidence documents were retrieved.
Method
GTA-RAG samples connected document paths from an entity–document graph, synthesizes and validates retrieval trajectories, then applies trajectory-guided GRPO followed by answer-reward training on natural QA.
Results
GTA-RAG consistently outperforms prior RL-based RAG systems across multi-hop and simple QA benchmarks with Qwen2.5-3B and Qwen2.5-7B, while improving complete evidence-chain coverage.
Takeaways & Limitations
Graph structure can provide scalable trajectory-level supervision for training more effective and evidence-grounded RAG agents.
Takeaways & Limitations
The framework depends on the quality of the entity–document graph and deployed retriever, and experiments focus on open-domain QA benchmarks.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) enables LLMs to access external knowledge for answering knowledge-intensive questions. For complex multi-hop questions, multi-turn retrieval-augmented reasoning extends RAG into an iterative process that repeatedly searches for and integrates evidence across documents. However, existing reinforcement-learning (RL) approaches for agentic RAG are typically optimized with final-answer rewards, which provide sparse supervision and overlook whether the model actually retrieves the required evidence chain. We present \textsc{GTA-RAG}, a graph-trajectory-augmented RL framework for multi-turn retrieval-augmented reasoning. From an entity--document graph, we sample connected document paths, synthesize multi-hop QA trajectories, and validate them with the deployed retriever to obtain executable trajectory-level supervision. We then optimize the retrieval policy with Group Relative Policy Optimization (GRPO) and a trajectory-guided reward that encourages both accurate answers and acquisition of target evidence documents, followed by answer-reward training on natural QA instances. Experiments on three multi-hop and two simple QA benchmarks show that \method{} consistently outperforms RL-based RAG baselines with both Qwen2.5-3B and Qwen2.5-7B backbones, while substantially improving evidence-chain coverage. Our code is available at https://github.com/cjcj46262/GTA-RAG.
1 Introduction
Multi-hop RAG requires evidence-grounded retrieval across connected documents, but answer-only RL rewards can reinforce unsupported shortcuts. GTA-RAG uses graph-derived executable trajectories and trajectory-guided GRPO to improve answer quality and evidence-chain acquisition.
- Multi-hop questions require identifying intermediate entities, issuing successive searches, and integrating evidence distributed across documents.
- Answer-level rewards may treat correct answers from incomplete or irrelevant retrieval as equivalent to evidence-grounded reasoning.
- GTA-RAG samples connected paths from an entity–document graph, synthesizes multi-hop questions, and validates executable retrieval trajectories.
- Its trajectory-guided GRPO objective jointly encourages accurate answers and faithful acquisition of documents along the evidence chain.
- GTA-RAG consistently improves multi-hop and simple QA performance while retrieval analyses associate gains with better evidence coverage rather than answer-only shortcuts.
2 Related Work
Related work contrasts similarity-based RAG and inference-time graph retrieval with GTA-RAG’s use of graph paths as executable supervision for intermediate retrieval behavior.
- Conventional RAG retrieves passages by lexical or dense similarity, which is less suited to evidence connected through intermediate entities or relations.
- Graph-based RAG organizes corpus units into structured representations and retrieves through graph connectivity.
- GTA-RAG uses graph paths to construct executable multi-document trajectories that supervise intermediate retrieval behavior, not only inference-time retrieval.
- Answer-level rewards provide limited supervision because correct answers can arise from parametric knowledge or incomplete evidence.
3 Method
GTA-RAG combines graph-structured retrieval, validated synthetic multi-hop trajectories, and two-stage GRPO training to teach evidence-grounded multi-turn reasoning.
- Framework: GTA-RAG uses an entity–document graph and a graph-augmented retriever as both the retrieval environment and source of trajectory-level supervision.The framework constructs heterogeneous graph representations and combines graph retrieval with dense passage retrieval.
- Graph retrieval: Graph retrieval propagates query relevance through relational and entity–document connections to identify structurally relevant documents beyond lexical similarity.The retriever retains top-Kg graph documents and combines them with top-Kd dense passage results after deduplication.
- Multi-turn rollout: The interactive policy repeatedly generates reasoning traces and either search queries or a final answer, updating its context with retrieved evidence after each search.Rollouts begin with only the question and terminate on an answer action or the maximum action budget.
- Trajectory construction: Connected document paths are converted into multi-hop questions and turn-by-turn trajectories whose intended support documents define candidate evidence chains.The generator conditions each turn only on the question and documents revealed earlier, matching policy information constraints.
- Trajectory validation: Every synthetic trajectory is executed through the deployed retriever and retained only when each designated support document appears at its corresponding search turn.This validation produces executable supervision rather than relying on graph connectivity alone.
- RL optimization: Two-stage GRPO first rewards acquiring new target documents and correct answers supported by complete evidence chains, then adapts the policy on natural QA instances with answer-level reward.The trajectory-guided objective supplies intermediate retrieval feedback before Stage II answer-reward training.
4 Experiments
GTA-RAG is evaluated across multi-hop and simple QA benchmarks, with controlled studies testing its components and retrieval behavior. It consistently improves RL-trained RAG agents, especially on multi-hop tasks, while recovering evidence chains more effectively.
- Evaluation Setup: The evaluation covers five open-domain QA benchmarks, including three multi-hop and two simple factual-retrieval datasets, using EM and token-level F1.Comparisons include vanilla, training-free, and RL-based RAG systems for Qwen2.5-3B and Qwen2.5-7B policies.
- Main Results: 44.5/53.6 and 50.5/59.3 average EM/F1 scores are achieved with Qwen2.5-3B and Qwen2.5-7B, respectively, outperforming the strongest RL-based alternatives.The advantage appears across both backbone groups rather than being tied to one model capacity.
- Main Results: 4.5 EM and 4.3 F1 points are gained over RouteRAG on multi-hop QA with Qwen2.5-3B, compared with 1.7 EM and 1.9 F1 points with Qwen2.5-7B.Simple-QA performance is competitive but not uniformly superior, matching the method’s focus on distributed evidence.
- Ablation Study: 74.1% to 58.7% full-chain coverage and a 3.4-point multi-hop F1 decrease result when trajectory-guided reward is removed.The ablation uses Qwen2.5-7B and examines trajectory-guided reward, trajectory validation, and graph retrieval under controlled training conditions.
- Ablation Study: Graph-derived trajectories are effective only when executable by the deployed retriever, while graph retrieval helps recover later-hop documents that dense retrieval may miss.Removing trajectory validation decreases coverage and answer quality; removing graph retrieval reduces multi-hop performance and evidence coverage.
- Retrieval Behavior Analysis: 82.3% target-document coverage and 74.1% complete-trajectory coverage are reached, versus 68.1% and 54.3% with answer-only RL.GTA-RAG also reduces zero-target turns while using slightly fewer searches, indicating more targeted retrieval rather than simply more retrieval calls.
5 Conclusion
GTA-RAG uses graph-sampled and validated retrieval trajectories with trajectory-guided GRPO to improve multi-turn retrieval-augmented reasoning. Experiments show consistent gains over RL-based RAG baselines and higher evidence-chain coverage, supporting graph structure as scalable supervision.
- Conclusion: GTA-RAG combines graph-sampled, validated retrieval trajectories with trajectory-guided GRPO for multi-turn retrieval-augmented reasoning.The framework encourages retrieval along required reasoning chains rather than answer-only shortcuts.
- Conclusion: Experiments on multi-hop and simple QA benchmarks show consistent improvements over strong RL-based RAG baselines.The reported gains are accompanied by higher target-document and complete-trajectory coverage.
- Conclusion: Graph structure is presented as scalable supervision for training more effective and evidence-grounded RAG agents.This conclusion follows from combining graph-derived trajectories with trajectory-guided reinforcement learning.
Limitations
GTA-RAG depends on the quality of its entity–document graph and deployed retriever, and its experiments focus on open-domain QA.
- Graph extraction noise or missing connections may limit the diversity of executable trajectories.
- The experiments focus on open-domain QA benchmarks, leaving broader domains and more diverse retrieval environments for future work.
A Additional Experimental Details
The paper reports statistics for validated synthetic trajectories and entity–document graphs used in training, retrieval, and trajectory construction.
- Table 4 summarizes the validated synthetic trajectories used for trajectory-guided reinforcement-learning training.
- Table 5 reports the scale of entity–document graphs constructed for each benchmark corpus.
- A larger HotpotQA-10K graph is constructed for trajectory augmentation during training.
B Prompt Templates
The prompt templates specify how graph-sampled document paths become executable multi-hop QA trajectories, including retrieval constraints, visibility rules, and structured outputs.
- The synthesis task decides whether candidate documents support a natural 2-hop, 3-hop, or 4-hop question.
- The template requires using only candidate documents and forbids revealing the answer in the question or search queries.
- The structured output records constructibility, question, gold answer, hop count, reasoning process, and per-turn search or answer actions.
- Questions must form a natural bridge, comparison, or entity-linking problem whose short answer is supported by selected documents.
- Search rounds use unique candidate documents, with later queries justified by previously retrieved evidence and no unnecessary documents.
- The first search query relies only on the question, while later turns expose the question and retrieved support documents.