SSRL: Self-Search Reinforcement Learning

• The study tests whether internal knowledge can simulate search and whether full-sim search RL transfers effectively to real web search.Source: §1 · p. 4
• The work quantifies and enhances Self-Search with SSRL, positioning LLMs as potential simulators for search-driven tasks without external queries.Source: §1 · p. 4
• LLMs can serve as world-knowledge simulators, with different model families exhibiting different upper bounds on challenging search-based benchmarks.Source: §1 · p. 4
• Reliably extracting the upper bound of Self-Search remains difficult, and voting is insufficient to select optimal outputs reliably.Source: §1 · p. 4
• SSRL using feedback from the policy model itself achieves superior performance to training with external search engines.Source: §1 · p. 4
• SSRL-trained models adapt to real search scenarios without additional effort, providing preliminary evidence of sim-to-real transfer.Source: §1 · p. 4

2.1 TASK FORMULATION

• pass@k estimates the internal search upper bound from K sampled answers; a fitted log-coverage relation models scaling with sample count.Correctness follows each benchmark’s evaluation standard, and the estimate is independent of external retrievers.Source: §2.1 · p. 5

2.2 PROMPT DESIGN

• The prompt elicits step-by-step thoughts, queries, and generated information in one autoregressive trajectory, continuing until the model produces a final answer.The question is appended to the prompt during training and inference.Source: §2.2 · p. 5, Table 1 · p. 5

2.4 PERFORMANCE EVALUATION

• Repeated sampling improves performance across benchmarks and model families; on BrowseComp, Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct surpass o1 with sufficient samples.On TriviaQA at 1024 samples, Llama-3.1-8B-Instruct scores 81.2% versus 81.4% for Llama-3.1-70B-Instruct.Source: Figure 2 · p. 6, §2.4 · pp. 6–7, Figure 3 · p. 7

2.3 EXPERIMENTAL SETUP

• Evaluation spans seven QA benchmarks, three task categories, and Qwen2.5, Llama3, and Qwen3 models from 0.6B to 72B under consistent sampling settings.Most benchmarks were constructed manually; 2WikiMultiHopQA and MuSiQue were exceptions.Source: §2.3 · pp. 5–6, Table 2 · p. 6

2.5.1 IS MORE REASONING ALWAYS BETTER?

• More thinking tokens, multi-turn self-search, and reflection do not improve performance over naive repeated sampling in the reported comparisons.The authors suggest knowledge utilization may be more advantageous than reasoning in these settings.Source: §2.5.1 · pp. 7–8

2.5.2 MAJORITY VOTING VS. PASS@K

• Increasing majority-voting sample count yields only marginal gains, leaving correct answers difficult to identify when ground truth is unknown.The authors note that incorrect answers may recur across samples.Source: §2.5.2 · pp. 8–9, Figure 6 · p. 9

3.1 TASK DEFINITION

• SSRL treats the policy as both reasoner and internal search engine, using GRPO and format and outcome rewards to train structured search trajectories.The outcome reward uses final-answer accuracy; the format reward enforces the prescribed reasoning structure.Source: §3.1 · p. 9, §3.2 · p. 10

3.4.1 SELF-SEARCH RL

• SSRL models consistently outperform models relying on external search engines and R1-like models on the reported benchmarks.Table 3 reports the main results across six benchmarks measured by EM.Source: §3.4.1 · p. 11, Table 3 · p. 11
• Training reward grows steadily; instruction-tuned models use internal knowledge more effectively, and Llama-3.1-8B-Instruct develops question decomposition and self-reflection.SSRL also takes 5.53× less training time than ZeroSearch and shows steady reward growth without collapse, despite lower early rewards.Source: §3.4.1 · pp. 11–12, Figure 8 · p. 12

3.4.2 SIM2REAL GENERALIZATION

• With format alignment maintained, SSRL-trained agents generally outperform Search-R1 and ZeroSearch while using less online searching in real-search settings.Performance improves with more maximum turns across models except Llama-3.1-8B-Instruct; entropy-guided selection reduces search frequency by 20–42% while maintaining comparable performance.Source: §3.4.2 · pp. 13–14, Table 4 · p. 13, Figure 9 · p. 13

3.4.3 TEST-TIME RL

• TTRL performs better than GRPO in the reported experiments, improves Llama-3.2-3B-Instruct average performance by 59%, and improves BrowseComp without external search.Without information masking, TTRL results are slightly better, contrary to RLVR.Source: §3.4.3 · p. 14, Table 7 · p. 14, Table 6 · p. 15

3.5.1 BENEFITS OF INFORMATION MASKING

• Information masking consistently improves performance across benchmarks, with authors attributing the gains to more detailed reasoning rather than copying retrieved information.The paper reports that masking information tokens encourages more comprehensive reasoning trajectories.Source: §3.5.1 · pp. 15–16, Figure 10 · p. 16

3.5.2 IMPACT OF FORMAT-BASED REWARD

• Format reward improves performance for base and instruction-tuned models across benchmarks and stabilizes training by limiting response-length growth.Without the reward, response length grows drastically.Source: §3.5.2 · pp. 16–17, Figure 11 · p. 16

3.5.3 IMPORTANCE OF ON-POLICY SELF-SEARCH

• Using the policy itself as the information provider avoids the collapse and performance degradation observed with a frozen provider; GRPO-based algorithms outperform PPO and REINFORCE++.The frozen-provider setting shows reward collapse after about 100 training steps; offline repeated rollouts can increase without additional cost.Source: §3.5.3 · p. 17, §3.5.4 · p. 18, Table 11 · p. 18

4.1 REINFORCEMENT LEARNING WITH SEARCH ENGINES

• Search-R1 and ReSearch use textual corpora, online-search RL incurs API costs, and ZeroSearch [Sun et al., 2025] simulates search.The upper bounds of LLM search world models for RL remain unknown.Source: §4.1 · p. 18

4.2 LARGE LANGUAGE MODELS AS SEARCH ENGINES

• LLM search-engine work includes document-identifier generation and responses summarizing retrieved items, but textual world models in agentic RL remain little explored.These approaches offer flexible information granularity and relevance matching compared with traditional retrieval systems.Source: §4.2 · pp. 18–19

4.3 INFERENCE-TIME SCALING OF LLMS AND AGENTS

• Repeated sampling improves the coverage of correct answers, but search applications remain underexplored; agent test-time scaling also uses parallel sampling, revision, backtracking, concurrent evaluation, and multi-agent verification.Repeated sampling generates multiple candidates from one prompt using probabilistic sampling.Source: §4.3 · p. 19

5 CONCLUSION

• SSRL enhances LLM self-search, outperforms search API-based baselines, and enables robust sim-to-real transfer.The authors describe this as a path toward more autonomous and scalable LLM agents that can operate without relying on external search engines.Source: §5 · p. 19

A.1.1 INSTRUCTIONS FOR REPEATED SAMPLING

• Repeated sampling uses the instruction in Table 12.The table is identified as the instruction for repeated sampling.Source: §A.1.1 · p. 24, Table 12 · p. 24

A.1.2 INSTRUCTIONS FOR LLM PROVIDING INFORMATION

• The LLM uses the instruction in Table 13 to provide information.The table is identified as the instruction for LLM information provision.Source: §A.1.2 · p. 24, Table 13 · p. 24

A.2 DETAILED RESULTS

• Figure 12 covers repeated sampling on seven benchmarks, while the appendix reports results across 16 models and fitting comparisons for Llama and Qwen models.Tables 15–17 compare actual and fitted values, residuals, and relative errors across model groups and K values; the supplied passages do not give cell values.Source: §A.2 · p. 24, Figure 12 · p. 25, Table 14 · p. 26

A.3.1 CASE STUDY FOR QWEN3 WITH/WITHOUT THINKING

• With Qwen3 thinking enabled, the model spends many tokens reasoning at the beginning.The case study is presented in Table 18.Source: §A.3.1 · p. 26

A.3.2 CASE STUDY FOR MULTI-TURN AND REFLECTION REPEATED SAMPLING

• Table 19 presents a case study of multi-turn and reflection-based repeated sampling.Source: §A.3.2 · p. 26

B.1 PROMPTS

• The appendix lists separate instructions for direct reasoning, RAG, and R1-like model training.These prompts are identified as Tables 20, 21, and 22, respectively.Source: §B.1 · p. 26

B.2.1 BASELINE IMPLEMENTATION

• Direct Answer and CoT use temperature 0.0, while RAG uses Jina and Google search at temperature 0.0.Source: §B.2.1 · p. 26
• R1 training uses GRPO, generates five responses per prompt, and uses batch size 256, learning rate 1e-6, and KL coefficient 0.001.Training runs up to five epochs with early stopping if response length or training accuracy collapses; evaluation uses temperature 0.0.Source: §B.2.1 · p. 27
• ZeroSearch uses the R1 setting, a two-turn maximum, and Google 14B as the simulation LLM.Its start and end thresholds are 0.0 and 0.5.Source: §B.2.1 · p. 27
• Search-R1 trains with e5 retrieval over Wikipedia and tests with Google Search.The passage says it otherwise uses the same setting as R1.Source: §B.2.1 · p. 27

B.2.2 OTHER ALGORITHM IMPLEMENTATION

• PPO, REINFORCE++, DAPO, and KL-Cov are trained for up to five epochs, with checkpoints selected by best performance and stopping on observed collapse.All use batch size 256 and learning rate 1e-6; PPO and REINFORCE++ use KL loss, while DAPO and KL-Cov do not.Source: §B.2.2 · pp. 27, 30

B.2.3 TTRL

• TTRL uses 32 rollouts per prompt at temperature 0.6, trains for 80 epochs, and stops when performance converges.The setup uses a learning rate of 5e-7, a 62-step warmup, and no KL loss.Source: §B.2.3 · p. 30

B.3.1 MODEL FAMILY COMPARISON

• Qwen’s absolute performance on General QA and Multi-Hop QA is lower than Llama’s, despite similar training patterns and Qwen’s strength in math and code tasks.The authors interpret this as weaker performance serving as a simulator of world knowledge.Source: §B.3.1 · pp. 30–31

B.3.2 COMPARISON BETWEEN GENERAL MODELS AND REASONING MODELS

• Qwen3 generally performs below Qwen2.5, and its search number can fall to zero during training amid instruction-following and format-reward concerns.The authors suggest that extensive math or code training may make reasoning models harder to transfer to other domains.Source: §B.3.2 · p. 31

B.3.3 DYNAMICS OF TRAINING WITH AND WITHOUT INFORMATION MASK

• The information mask significantly increases search behavior activity, potentially improving reasoning capabilities on complex tasks.The figure compares training dynamics with and without the mask.Source: §B.3.3 · p. 31

B.3.4 GROUP SIZE ABLATION

• Larger group sizes achieve comparable performance while converging faster.The ablation uses Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct; Figure 14 compares group sizes and Table 25 reports their performance.Source: §B.3.4 · pp. 31–32, Figure 14 · p. 32, Table 25 · p. 32

B.3.5 ADDITIONAL BENCHMARKS

• On SimpleQA, internal knowledge alone does not help complete tasks, while external knowledge access shows potential for organizing queries and reasoning over gathered information.The evaluation samples 200 records and considers offline search, online search, and entropy-guided search.Source: §B.3.5 · p. 32

B.4 CASE STUDIES

• Case studies cover SSRL-trained Llama-3.2-3B and Qwen2.5-7B on General and Multi-hop QA, real-search integration, and TTRL-trained Llama on BrowseComp.Source: §B.4 · p. 32, Table 28 · p. 34, Table 29 · p. 35

B.3.6 ADDITIONAL RESULTS FOR SIM2REAL SEARCH

• First-search information anchors successful trajectories; Table 26 compares replacing retrieved information at the first versus last search step with real search.The experiment tests the importance of the first search and its information by replacing information from either step and regenerating the answer.Source: §B.3.6 · p. 32, Table 26 · p. 33

C FORMAT REWARD CODE

• Format reward validates allowed tags, nonempty contents, and search-information pairing between initial thought and final answer.Source: C FORMAT REWARD CODE · pp. 33, 38