Multi-hop Reasoning via Early Knowledge Alignment

• Iterative RAG addresses multi-hop retrieval through repeated reasoning and search, but an ungrounded initial plan can lead to poor retrieval decisions and cascading errors.Source: §1 · pp. 1–2, Figure 1 · p. 2
• EKA augments the initial thinking step with early knowledge, grounding models before they enter the RL-optimized iterative retrieval and generation process.The paper says this improves reasoning foundations and reduces the likelihood of cascading errors in later iterations.Source: §1 · p. 2
• The entropy analysis presents lower training entropy as more efficient exploration focused on the retrieval set, rather than insufficient exploration.It contrasts this with uninformed, model-initialized thinking during RL training.Source: §1 · p. 2
• EKA reports consistent improvements in answer accuracy and retrieval recall across standard RAG datasets, with no degradation in generalization experiments.Source: §1 · p. 2

2.1 Retrieval-Augmented Generation

• Dense Passage Retrieval and RAG established retrieval-augmented open-domain QA, while later work developed dual-level and structure-based retrieval methods.Source: §2.1 · p. 2

2.2 Iterative and Multi-Hop RAG Approaches

• Iterative RAG approaches interleave reasoning and retrieval or adapt retrieval timing using confidence, reflection, or low-confidence signals.Examples include IRCoT, ITER-RETGEN, Self-RAG, Selfask, and FLARE.Source: §2.2 · p. 3

2.3 Reinforcement Learning for RAG Optimization

• RL-for-RAG systems use SFT followed by RL or apply RL directly; direct-RL approaches may produce redundant paths when models are not aligned with the retrieval set.Source: §2.3 · p. 3
• EKA applies early knowledge alignment to address the retrieval-set misalignment problem identified for direct-RL approaches.Source: §2.3 · p. 3

3.1 PPO

• PPO fine-tuning maximizes a clipped policy objective using the ratio between current- and old-policy token probabilities.The clipping parameter constrains the ratio to [1 − ϵ, 1 + ϵ], and the advantage is typically computed using GAE.Source: §3.1 · p. 3

3.2 GRPO

• GRPO avoids value-function approximation by using average rewards from multiple sampled outputs as a baseline, with advantages computed from relative rewards within each group.Source: §3.2 · pp. 3–4

4 Method

• EKA gives the policy model early retrieved knowledge before its first thinking step, then follows the standard GRPO rollout and update phases.The module is intended to address planning without sufficient contextual grounding, which can lead to suboptimal retrieval and redundant exploration.Source: §4 · p. 4

4.1 Early Knowledge Alignment

• The initial retrieval step gathers the top-k relevant passages from corpus D for question q, forming P0 before initial thinking.P0 is defined as the set of initially retrieved passages.Source: §4.1 · p. 4

4.2 Iterative Thinking and Searching

• After the initial search, the model iterates through thinking, searching, and answering actions until it generates a final answer.Thinking uses current knowledge; searching queries the knowledge base, and answering occurs when sufficient information is gathered.Source: §4.2 · p. 4

4.3 Theoretical Analysis

• The paper’s proposition states that EKA is better than traditional iterative-RAG thinking from an entropy perspective.The formal proof is in Appendix C, with empirical entropy results reported in Section 6.1.Source: §4.3 · p. 4

5.1 Implementations

• Experiments use different dataset splits and retrieval corpora across Search-R1 and Graph-R1, with six datasets in Graph-R1 and Bamboogle added in Search-R1.The evaluation uses EM, F1, and R-S in Graph-R1 and F1 in Search-R1.Source: §5 · pp. 4–5, §5.1 · pp. 5–6

5.2 Comparison in Graph-R1 Setting

• Graph-R1 improves by an average of 3 F1 points with EKA in the Graph-R1 setting.Table 2 reports the main results in that setting; the authors also report average gains of 11 F1 points for Search-R1 and 7 for Search-R1-PPO.Source: Table 2 · p. 5, §5.2 · p. 6

5.1 Implementations

• The experiments use Qwen2.5-7B and Qwen2.5-14B backbones, with E5 retrieval for Search-R1 and hypergraph retrieval for Graph-R1.The baseline comparisons include training-free and trained methods in Graph-R1, with additional baselines in Search-R1.Source: §5.1 · p. 5

5.2 Comparison in Graph-R1 Setting

• Search-R1 improves by an average of 11 F1 points and Search-R1-PPO by an average of 7 F1 points with EKA.The authors ran Search-R1-PPO themselves because Search-R1 uses PPO while Graph-R1 uses GRPO.Source: §5.2 · p. 6
• EKA’s higher R-S scores compared with Graph-R1 suggest that performance gains are partially driven by improved retrieval quality.The authors interpret the R-S improvement as better exploitation of retrieval-relevant information.Source: §5.2 · p. 6

5.3 Comparison in Search-R1 Setting

• In the Search-R1 setting, EKA improves results with Fullwiki retrieval across in-domain and out-of-domain datasets by an average of 6.3 F1 points.The authors use Search-R1 alone because constructing a full-Wikipedia hypergraph in the Graph-R1 manner is computationally prohibitive.Source: §5.3 · p. 6

5.4 Training-free EKA

• As a training-free inference module on larger models, EKA delivers substantial benchmark gains and mitigates hallucinations and improves reasoning stability without parameter updates.The authors evaluate this setting because RL fine-tuning is computationally prohibitive for the larger models.Source: §5.4 · p. 6

6.1 Entropy Analysis

• EKA generally lowers answer, think, and query-token entropy, with one training step as an exception for answer-token entropy.The authors interpret lower think and search entropy as a more determined exploration direction.Source: §6.1 · pp. 6–7, Figure 3 · p. 6

6.2 Shorter turns and Metrics Dynamics.

• EKA shortens exploration by about one turn on average, while its R-S score is high from the start and remains higher when early knowledge is excluded.Figure 4 also tracks F1 and R-S across training steps.Source: §6.2 · p. 7, Figure 4 · p. 7, Table 6 · p. 7

6.3.1 Generalization across datasets

• With the Graph-R1 backbone, EKA performs better on average in both IID and OOD generalization tests.The test uses training datasets as table rows and test datasets as columns.Source: §6.3.1 · p. 7, Table 7 · p. 8

6.3.2 Mismatched Early Knowledge

• Noisy early knowledge from the full Wikipedia corpus slightly reduces performance versus standard EKA, but still beats the no-EKA baseline on average.The authors attribute the remaining benefit to early knowledge’s grounding effect, even when that knowledge is imperfect.Source: §6.3.2 · p. 8, Table 8 · p. 8
• EKA yields consistent performance gains with both the default BGE retriever and the alternative E5 retriever across six datasets.The authors characterize the framework as retriever-agnostic across different semantic embedding spaces.Source: §6.3.2 · p. 8, Table 9 · p. 8

7 Conclusion

• EKA guides thinking in iterative RAG toward more efficient RL exploration and better end-to-end performance across Search-R1 and Graph-R1, PPO and GRPO, and varied retrieval contexts.The conclusion includes small structured corpora and large unstructured document sets among the tested retrieval contexts.Source: §7 · p. 8
• EKA maintains or improves backbone generalization and also scales as a plug-and-play, training-free module for large models.The conclusion frames this as a shift from plan-first design toward early knowledge alignment.Source: §7 · pp. 8–9

8 Limitations

• Whether EKA works in more complex Deepresearch scenarios remains unknown.The stated limitation bounds the reported evidence to multi-hop QA.Source: §8 · p. 9

9 Reproducibility Statement

• Training details, technical proofs, and additional experimental and implementation details are provided in Appendices A, C, and D, respectively, with code supplied as supplemental material.Source: §9 · p. 9

A Algorithm

• The algorithm retrieves initial knowledge P0 from the retrieval set and appends it to the input before the iterative process.It then generates tokens within a bounded loop, searches when a query is produced, and stops on an answer or end condition.Source: A Algorithm · p. 12

B.1 Qwen3 Model Results

• Even with poor results, EKA improves Qwen3 performance; the authors attribute low performance to difficulty generating the pipeline’s thinking process after removal of the think pattern.They connect this difficulty to Qwen3 instruction models’ pre-training with a think token.Source: §B.1 · p. 12

B.2 Case Study

• Without planning to split the question, Graph-R1 repeatedly searches for both items and fails to retrieve the directors.The passage identifies this as a case where the model lacks the planning ability to divide the question into two parts.Source: §B.2 · p. 12
• With EKA, the model searches for the two directors separately, retrieves the right documents, and reaches the right answer.The passage contrasts this outcome with the unsuccessful Graph-R1 search in Table 10.Source: §B.2 · p. 12

C Theoretical Proof

• EKA is no worse than traditional thinking from an entropy perspective and is argued to improve information gain and lower entropy, supporting more efficient and accurate question answering.The proof frames greater entropy reduction under the same budget, or fewer tokens to reach the same posterior certainty, as greater efficiency; lower conditional entropy corresponds to lower expected error.Source: C Theoretical Proof · pp. 12–14
• The proof models iterative RAG as evidence gathered over rounds, with each new retrieval selected using the current belief and uncertainty.A Lyapunov-style potential Vt represents remaining uncertainty after round t.Source: C Theoretical Proof · pp. 12–13
• Figure 5 reports Qwen3-4B-Instruct-2507 F1 scores at each step on 2Wiki using Graph-R1 as the backbone.Source: Figure 5 · p. 13

D.1 Baselines in Graph-R1 Setting

• The Graph-R1 setting compares GPT-4o-mini-based NaiveGeneration and StandardRAG with graph-based retrieval methods, alongside Qwen2.5-Instruct (7B) and RL-trained retrieval baselines.The listed graph methods include GraphRAG, LightRAG, PathRAG, HippoRAG2, and HyperGraphRAG; the RL-based set includes Search-R1, R1-Searcher, and Graph-R1.Source: §D.1 · pp. 14–15

D.2 Baselines In Search-R1 Setting

• The Search-R1 setting adds CoT, IRCoT, Search-o1, and Rejection Sampling as reasoning and generation comparisons.Source: §D.2 · p. 15

D.3 Metrics

• Evaluation includes exact-match answer accuracy, token-overlap F1, and semantic similarity between retrieved and gold contexts.Exact Match uses normalized answers; F1 balances token precision and recall, while Retrieval Similarity compares semantic embeddings.Source: §D.3 · pp. 15–16

D.4 Hyperparameters

• In the Graph-R1 setting, hyperparameters are reported in Table 12, and EKA models share the backbone method’s hyperparameters.Source: §D.4 · p. 16, Table 12 · p. 16