Source-linked AI summary
SEM: Reinforcement Learning for Search-Efficient Large Language Models
Zeyang Sha, Shiwen Cui, Weiqiang Wang
TL;DR
Models can struggle to decide when search is necessary, and reinforcement learning can still produce redundant searches. SEM combines balanced training data and a search-aware reward policy; across evaluated benchmarks, the authors report reduced redundant search and improved answer accuracy.
Problem
Teaching models to distinguish when to search from when to rely on internal knowledge remains challenging, and existing reinforcement learning approaches can produce unnecessary searches.
Source: §1 · pp. 1–2Method
SEM post-trains models using MuSiQue and MMLU, with a reward policy that evaluates first-answer correctness, search use, and response formatting.
Source: §2.1 · p. 3, §2.2 · p. 4Results
Across HotpotQA, MuSiQue, MMLU, and GSM8K, SEM reduces redundant search while improving answer accuracy; on MMLU, Qwen2.5-7B-Instruct achieves 70.88 EM versus Naive RAG’s 12.48 EM.
Source: §3.2 · p. 6, §5 · p. 9Takeaways & Limitations
The authors conclude that SEM improves models’ ability to discern when external retrieval is beneficial and promotes structured response protocols.
Source: §3.2 · p. 6Under the reported experimental setup, the training-dataset composition makes ReSearch models prone to gradient explosion, leading them to overuse search or fail to use it effectively.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Recent advancements in Large Language Models(LLMs) have demonstrated their capabilities not only in reasoning but also in invoking external tools, particularly search engines. However, teaching models to discern when to invoke search and when to rely on their internal knowledge remains a significant challenge. Existing reinforcement learning approaches often lead to redundant search behaviors, resulting in inefficiencies and over-cost. In this paper, we propose SEM, a novel post-training reinforcement learning framework that explicitly trains LLMs to optimize search usage. By constructing a balanced dataset combining MuSiQue and MMLU, we create scenarios where the model must learn to distinguish between questions it can answer directly and those requiring external retrieval. We design a structured reasoning template and employ Group Relative Policy Optimization(GRPO) to post-train the model's search behaviors. Our reward function encourages accurate answering without unnecessary search while promoting effective retrieval when needed. Experimental results demonstrate that our method significantly reduces redundant search operations while maintaining or improving answer accuracy across multiple challenging benchmarks. This framework advances the model's reasoning efficiency and extends its capability to judiciously leverage external knowledge.
1 Introduction
LLMs can use search to answer unfamiliar questions, but learning when to search—including when to search again—remains difficult, and reinforcement learning can still prompt unnecessary searches. SEM trains models to distinguish questions answerable from internal knowledge from those needing retrieval.
1 Introduction
- Models struggle to recognize errors in initial searches and initiate subsequent searches during iterative interactions.Source: §1 · pp. 1–2
- Existing reinforcement learning approaches can prompt unnecessary searches, including multiple queries for a trivial question like “1+1=?”.The paper identifies efficient search decisions as a way to avoid wasted resources.Source: §1 · p. 2
1.1 Our Contribution
- SEM trains models to answer directly when confident and search when uncertain, using a balanced dataset and rewards for appropriate search behavior.When an initial answer is correct, subsequent search is penalized; when it is incorrect, a search query is rewarded.Source: §1.1 · p. 2
- The authors report that this training reduces redundant actions and improves response precision and describe search-awareness as supporting resource-efficient handling of complex, uncertain questions.Source: §1.1 · p. 2
2 Method
SEM post-trains a model to decide when external search is useful and how to incorporate retrieved information. It combines a balanced known/unknown-question dataset with GRPO, a reward policy for answer correctness, search decisions, and response format, and a structured reasoning template.
2 Method
- SEM post-trains the base model with reinforcement learning to integrate external search into its reasoning.Figure 1 illustrates the methodology.Source: §2 · p. 2
2.1 Dataset Preparation
- MuSiQue [14] supplies multi-hop fact questions often beyond pretraining knowledge, while MMLU [15] supplies questions generally covered by existing corpora; together they balance known and unknown questions.The training distribution supports different feedback for correct answers without search and cases where search is needed.Source: §2.1 · p. 3
2.2 Reward Policy
- GRPO [16] compares output quality within each query group, while the reward policy favors correct answers without needless search, strategic search when needed, and correctly ordered response tags.An initially correct answer above an F1 threshold is penalized for unnecessary search or redundant reasoning; an incorrect first answer requires a search and a second answer, while invalid formatting receives zero reward.Source: §2.2 · pp. 3–4
- During rollout, a search query is followed by retrieved results, updated reasoning, and a final answer, allowing the method to assess search’s contribution to accuracy.Source: §2.2 · p. 4
2.3 Training Template
- The training template structures outputs with reasoning, preliminary and final answers, and optional search and retrieved-result fields to support parsing, reward evaluation, and targeted feedback.Source: §2.3 · p. 4
3 Experiments
Experiments train SEM on MuSiQue and MMLU, then evaluate it across four benchmarks covering questions that may or may not require external search. The reported results combine improved benchmark performance with selective search, steadier training curves, and case-study examples of direct answering or focused retrieval.
3.1 Experimental Setup
- SEM training combines MuSiQue, generally unfamiliar questions, with MMLU, generally within existing model knowledge, then evaluates on four benchmarks.Evaluation uses MuSiQue, HotpotQA, MMLU, and GSM8K; the setup treats search ratio differently across these task types.Source: §3.1 · pp. 4–5
3.2 Main Results
- 35.84 EM on HotpotQA vs Naive RAG’s 18.01 and 15.59 EM on MuSiQue vs 7.19; SEM is reported superior across evaluated benchmarks.The reported comparisons are for Qwen2.5-7B-Instruct.Source: §3.2 · p. 5, Table 1 · p. 5
- 1.77% SR on MMLU vs Naive RAG’s 47.98% accompanies 70.88 EM vs 12.48; on GSM8K, 79.37% EM vs ReSearch’s 50.41%.The GSM8K passage notes that the model was trained only on MMLU.Source: §3.2 · p. 6, Table 2 · p. 6
3.3 Training Process
- Across 7B and 14B training, SEM’s average F1 exceeds ReSearch with smoother, steadier gains; curves average 100 evaluation samples per checkpoint.The reported advantage is more pronounced for the 14B model.Source: §3.3 · pp. 6–7, Figure 2 · p. 7
3.4 Case Study
- In case studies, SEM answers known 1+1 directly and uses fewer, focused searches than ReSearch for the unknown NHL-draft-city question.The examples illustrate avoiding retrieval when internal knowledge suffices while retaining multi-hop retrieval for an external-knowledge question.Source: §3.4 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8
4 Related Work
The section reviews reinforcement-learning methods for post-training LLMs and agent frameworks for multi-step tasks. It contrasts fixed or heuristic tool management with SEM’s explicit RL training of tool-use patterns.
4.1 Reinforcement Learning
- PPO limits policy deviation during updates to balance exploration and exploitation, while DPO optimizes model parameters from preference data without a separate reward model.DPO uses a classification loss to encourage preference-aligned responses.Source: §4.1 · pp. 8–9
- GRPO compares groups of outputs to derive relative advantages and has shown promise for improving LLM reasoning on complex tasks such as mathematical problem-solving.The section frames RL methods as tools for post-training refinement of coherence, preference alignment, and reasoning.Source: §4.1 · p. 9
4.2 Large Language Models as Agents
- Agent frameworks such as AutoGPT and LangChain iteratively refine tasks, search for information, and generate solutions, while SEM trains tool-use patterns through RL rather than relying on heuristics or fixed prompts.Prior work also emphasizes tool selection and timing [11].Source: §4.2 · p. 9
5 Conclusion
SEM is a post-training reinforcement-learning framework that uses a balanced dataset and a reward function to optimize search behavior. Experiments report reduced redundant searches alongside enhanced answer accuracy.
5 Conclusion
- SEM’s balanced dataset distinguishes known from unknown questions, and its reward function penalizes unnecessary searches while encouraging effective retrieval.Source: §5 · p. 9
- SEM significantly reduces redundant search operations while enhancing answer accuracy across evaluations on HotpotQA, MuSiQue, MMLU, and GSM8k.The model is trained on a dataset combining MuSiQue and MMLU.Source: §5 · p. 9
- The reported experiments assess a model trained on combined MuSiQue and MMLU data across HotpotQA, MuSiQue, MMLU, and GSM8k.Source: §5 · p. 9