PaperlayerView exact revision on arXiv

Source-linked AI summary

Search-o1: Agentic Search-Enhanced Large Reasoning Models

Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, Zhicheng Dou

arXiv:2501.05366v1cs.AIcs.CLcs.IR

TL;DR

Large reasoning models’ extended reasoning can expose knowledge gaps and uncertainty, motivating methods for autonomous knowledge supplementation. Search-o1 combines agentic RAG with a Reason-in-Documents module to retrieve and refine external knowledge during reasoning. Across complex reasoning tasks and open-domain QA benchmarks, it reports stronger performance and potential improvements in LRM reliability and versatility.

  • Problem

    Extended reasoning in LRMs can encounter knowledge insufficiency, allowing gaps to propagate errors and disrupt the reasoning chain.

  • Method

    Search-o1 integrates agentic RAG and a Reason-in-Documents module that retrieves knowledge on demand and refines documents before reasoning-chain integration.

  • Results

    Search-o1 outperforms RAgent-QwQ-32B and QwQ-32B by 4.7% and 3.1% on average across five datasets, respectively.

  • Takeaways & Limitations

    The experiments support Search-o1’s potential to improve the reliability and versatility of LRMs for complex problem solving.

Abstract

from arXiv · show

Large reasoning models (LRMs) like OpenAI-o1 have demonstrated impressive long stepwise reasoning capabilities through large-scale reinforcement learning. However, their extended reasoning processes often suffer from knowledge insufficiency, leading to frequent uncertainties and potential errors. To address this limitation, we introduce \textbf{Search-o1}, a framework that enhances LRMs with an agentic retrieval-augmented generation (RAG) mechanism and a Reason-in-Documents module for refining retrieved documents. Search-o1 integrates an agentic search workflow into the reasoning process, enabling dynamic retrieval of external knowledge when LRMs encounter uncertain knowledge points. Additionally, due to the verbose nature of retrieved documents, we design a separate Reason-in-Documents module to deeply analyze the retrieved information before injecting it into the reasoning chain, minimizing noise and preserving coherent reasoning flow. Extensive experiments on complex reasoning tasks in science, mathematics, and coding, as well as six open-domain QA benchmarks, demonstrate the strong performance of Search-o1. This approach enhances the trustworthiness and applicability of LRMs in complex reasoning tasks, paving the way for more reliable and versatile intelligent systems. The code is available at \url{https://github.com/sunnynexus/Search-o1}.

1 Introduction

Large reasoning models can develop knowledge gaps during extended reasoning, creating uncertainty and error-propagation risks. Search-o1 addresses this by retrieving knowledge on demand and refining retrieved documents before integrating them into the reasoning chain.

  • Motivation: Extended reasoning can produce knowledge gaps whose errors propagate through the reasoning chain.The paper links long internal reasoning chains with knowledge insufficiency and disrupted reasoning.
  • Motivation: Over 30 occurrences of “perhaps” per reasoning process indicate frequent uncertainty on challenging GPQA diamond problems.The analysis reports numerous uncertain terms and notes that specialized problems make manual verification costly.
  • Search-o1: Search-o1 integrates agentic search into LRM reasoning to retrieve external knowledge autonomously when knowledge shortages arise.Its retrieval workflow can be triggered and iterated during a single reasoning process.
  • Search-o1: Reason-in-Documents analyzes retrieved documents using the current query and prior reasoning steps, then produces refined information for integration.The module operates independently from the main reasoning chain to reduce redundancy and preserve coherence.
  • Contributions: Search-o1 combines agentic RAG and knowledge refinement, with experiments spanning five complex reasoning domains and six open-domain QA benchmarks.The evaluation also reports quantitative analyses of efficiency and scalability.

2 Related Work

Related work covers large reasoning models, methods for developing o1-like reasoning, and retrieval-augmented generation. Agentic RAG enables autonomous retrieval decisions, but prior RAG approaches had not combined this capability with o1-like reasoning.

  • Large Reasoning Models: Large reasoning models improve test-time performance through extended reasoning steps rather than scaling model size or training data.Studies also report improved complex-task reasoning for smaller models through test-time scaling.
  • Large Reasoning Models: Prior approaches develop o1-like reasoning through MCTS, deliberate reasoning errors, or distilled training data.These methods differ in whether reasoning is internalized within the model.
  • Retrieval-Augmented Generation: RAG addresses limitations of static generative-model parameters by providing external knowledge through retrieval mechanisms.Research has advanced query preprocessing, document compression, denoising, and refinement.
  • Retrieval-Augmented Generation: Agentic RAG lets models autonomously decide when and what knowledge to retrieve, supporting planning and problem solving.Existing RAG approaches had not combined these capabilities with the reasoning strength of o1-like models.

3 Methodology

Search-o1 integrates autonomous retrieval and document refinement into LRM reasoning, allowing external knowledge to be retrieved on demand and incorporated as concise information while preserving reasoning coherence.

  • 3.1 Problem Formulation: Search-o1 maps task instructions, questions, and dynamically retrieved documents to a reasoning chain and final answer.The formulation represents the problem-solving objective as (I, q, D) → (R, a).
  • 3.3 Agentic Retrieval-Augmented Generation Mechanism: The agentic RAG mechanism lets the model decide during reasoning when to generate a search query and retrieve relevant documents.Search queries are conditioned on the current reasoning state and previously retrieved knowledge; retrieval then resumes the reasoning process.
  • 3.3 Agentic Retrieval-Augmented Generation Mechanism: Retrieved documents are injected into the accumulated reasoning chain so the model can continue reasoning with dynamically acquired external knowledge.The documents are inserted between search-result delimiters after the search function returns the top-k_i relevant documents.
  • 3.4 Knowledge Refinement via Reason-in-Documents: Reason-in-Documents analyzes retrieved documents separately using the prior reasoning chain and current query, then produces refined knowledge for the main chain.This separate process addresses lengthy, redundant, and potentially coherence-disrupting retrieved content.
  • 3.5 Search-o1 Inference: The inference algorithm alternates batch reasoning, search-query extraction, document retrieval, document reasoning, and reinsertion until sequences finish.Algorithm 1 initializes unfinished and finished sequence sets and outputs finished sequences after iterative processing.

4 Experiments

The experiments evaluate Search-o1 across challenging science, mathematics, coding, and open-domain QA benchmarks, comparing direct reasoning, standard RAG, agentic RAG, and human experts. Search-o1 generally improves performance through agentic retrieval and Reason-in-Documents, while scaling analysis examines retrieved-document usage.

  • Tasks and Datasets: The evaluation covers five challenging reasoning domains and six open-domain QA benchmarks.The challenging tasks include PhD-level science QA, mathematics, and coding; open-domain QA includes single-hop and multi-hop datasets.
  • Baselines and Settings: The baselines include direct reasoning, standard RAG, and RAG Agent methods, with experiments using QwQ-32B-Preview as Search-o1’s backbone.Retrieval uses Bing Web Search and top-k document retrieval, while standard RAG retrieves top-10 documents for the original question.
  • Main Results: QwQ-32B-Preview outperforms traditional instruction-tuned and larger LLMs in direct reasoning across the challenging reasoning tasks.The 32B-parameter QwQ model surpasses Qwen2.5-72B and Llama3.3-70B in the direct reasoning setting.
  • Main Results: Search-o1 exceeds RAgent-QwQ-32B and QwQ-32B by 4.7% and 3.1%, respectively, averaged across five challenging reasoning datasets.It also exceeds Qwen2.5-32B and Llama3.3-70B by 44.7% and 39.3%, respectively.
  • Scaling Analysis: Search-o1 improves as more documents are retrieved, and one retrieved document can outperform direct reasoning and standard RAG using ten documents.The analysis attributes this pattern to the agentic search and Reason-in-Documents strategies.
  • Human-Expert Comparison: On the GPQA extended set, Search-o1 scores 57.9 overall, outperforming human experts in physics and biology but trailing chemists in chemistry.The reported domain scores are 68.7 in physics, 69.5 in biology, and 40.7 versus 72.6 in chemistry.
  • Open-Domain QA: On open-domain QA, agentic RAG improves QwQ-32B’s average EM over standard RAG by 23.2% on multi-hop tasks, while single-hop performance changes little.The reported single-hop averages are 47.8 versus 47.6 EM.

5 Conclusion

Search-o1 addresses knowledge insufficiency in large reasoning models by combining agentic retrieval-augmented generation with a Reason-in-Documents module. Across complex reasoning and open-domain QA benchmarks, it consistently outperforms direct reasoning and existing retrieval-augmented methods, with performance comparable to or exceeding human experts in specific domains.

  • Search-o1 integrates agentic retrieval-augmented generation with a Reason-in-Documents module to address knowledge insufficiency in LRMs.The framework autonomously retrieves and incorporates external knowledge during reasoning while refining retrieved documents.
  • Search-o1 consistently outperforms existing retrieval-augmented and direct reasoning methods across science, mathematics, coding, and open-domain QA tasks.The evaluation covers diverse complex reasoning tasks and multiple open-domain QA benchmarks.
  • Search-o1 achieves performance comparable to or exceeding human experts in specific domains.

A Instruction Templates

The appendix defines instruction templates for Search-o1 and its Reason-in-Documents process, including search-result handling, fallback outputs, and iterative reasoning with retrieved information.

  • A.1 Instructions for Search-o1: The appendix labels the Search-o1 instruction separately from the Reason-in-Documents instruction.
  • A.1 Instructions for Search-o1: The Search-o1 instruction introduces a reasoning assistant that can perform web searches using delimited search-query and search-result markers.
  • A.1 Instructions for Search-o1: The assistant may repeat searches when necessary, while the system returns processed information from relevant web pages for continued reasoning.
  • A.1 Instructions for Search-o1: The Reason-in-Documents instruction asks the model to analyze web pages using previous reasoning steps, the current search query, and searched pages.
  • A.1 Instructions for Search-o1: The document-analysis task prioritizes factual information relevant to the current query and useful for advancing the original reasoning process.
  • A.1 Instructions for Search-o1: When pages provide helpful information, the output must begin with the specified ‘Final Information’ format.
  • A.1 Instructions for Search-o1: When no helpful information is found, the template outputs a fallback message containing previous reasoning steps, the search query, and searched web pages.
  • A.1 Instructions for Search-o1: The template explicitly directs the model to analyze each web page for information relevant to the current search query and previous reasoning steps.

A.2 Instructions for Standard RAG

The Standard RAG template instructs an assistant to answer a question using supplied documents by extracting relevant information and synthesizing a coherent response.

  • Standard RAG provides the assistant with a question and a set of documents as answer-generation inputs.
  • The assistant must analyze the documents to extract information relevant to the user’s question.
  • The final response should synthesize document information into a coherent and accurate answer that directly addresses the question.

A.3 Instructions for RAG Agent

The appendix specifies separate templates for an iterative RAG agent and for open-domain QA, mathematics, multiple-choice, and coding tasks, including their input formats and output constraints.

  • A.3 Instructions for RAG Agent: The RAG Agent can issue web-search queries through delimited markers and receive search results containing webpage titles, URLs, and snippets.
  • A.3 Instructions for RAG Agent: The agent can request full page content for selected URLs after receiving search results.
  • A.3 Instructions for RAG Agent: The RAG Agent supports repeated searches but limits search attempts and the number of URLs fetched for detailed information.
  • A.4.1 Open-Domain QA Tasks Instruction: Open-domain QA tasks require the final answer to follow the boxed-answer format.
  • A.4.2 Math Tasks Instruction: Math tasks use a dedicated prompt and likewise require the final answer in boxed form.
  • A.4.3 Multi-choice Tasks Instruction: Multiple-choice tasks require selecting one of A, B, C, or D and returning only the boxed choice.
  • A.4.4 Code Tasks Instruction: Code tasks require a correct Python program that passes all tests and is returned inside a Python code block.
  • A.4.4 Code Tasks Instruction: The code-task template supplies separate fields for the problem title and problem statement.

A.5 Additional Notes

The instructions distinguish prompt settings for reasoning and non-reasoning models. QwQ-32B-Preview receives task-specific instructions, while other listed models receive an explicit step-by-step prompt.

  • QwQ-32B-Preview receives task-specific instructions as user prompts rather than system prompts.
  • Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct, and Llama3.3-70B-Instruct receive an added prompt requesting step-by-step reasoning before answering.

B Case Study

The case-study tables illustrate Search-o1 outputs across GPQA, AMC2023, and HotpotQA. Search queries and refined results are specially delimited, while Reason-in-Documents integrates coherent information into the ongoing reasoning process.

  • Tables 4, 5, and 6 show Search-o1 outputs on GPQA, AMC2023, and HotpotQA, respectively.
  • Search-generated queries are enclosed by <|begin_search_query|> and <|end_search_query|> markers.
  • Refined search results are enclosed by <|begin_search_result|> and <|end_search_result|> markers.
  • Reason-in-Documents supplies coherent information that meets the current reasoning step's needs and integrates with preceding reasoning.
Loading 2501.05366v1…