PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Deep Search: Democratizing Search with Open-source Reasoning Agents

Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, Himanshu Tyagi, Pramod Viswanath

arXiv:2503.20201v1cs.LGcs.CLcs.IR

TL;DR

Proprietary search AI products dominate while closed systems limit transparency, innovation, and entrepreneurship. ODS combines an open web-search tool with a reasoning agent that can use a user-chosen LLM; with DeepSeek-R1, it surpasses GPT-4o Search Preview on FRAMES and nearly matches it on SimpleQA.

  • Problem

    Proprietary search AI solutions dominate, while closed-source products limit transparency, innovation, and entrepreneurship.

    Source: §1 · p. 1

  • Method

    ODS pairs an Open Search Tool that retrieves and filters web context with an Open Reasoning Agent that interprets queries and selects tools, including search.

    Source: §1 · p. 2

  • Results

    With DeepSeek-R1, ODS achieves 75.3% accuracy on FRAMES, surpassing GPT-4o Search Preview by 10%, and ODS-v2 achieves 88.3% on SimpleQA.

    Source: §3.1.2 · p. 9, §4 · p. 12

  • Takeaways & Limitations

    ODS is a plug-and-play open-source framework that can pair search and reasoning capabilities with a user-chosen LLM.

    Source: §4 · pp. 11–12

Abstract

from arXiv · show

We introduce Open Deep Search (ODS) to close the increasing gap between the proprietary search AI solutions, such as Perplexity's Sonar Reasoning Pro and OpenAI's GPT-4o Search Preview, and their open-source counterparts. The main innovation introduced in ODS is to augment the reasoning capabilities of the latest open-source LLMs with reasoning agents that can judiciously use web search tools to answer queries. Concretely, ODS consists of two components that work with a base LLM chosen by the user: Open Search Tool and Open Reasoning Agent. Open Reasoning Agent interprets the given task and completes it by orchestrating a sequence of actions that includes calling tools, one of which is the Open Search Tool. Open Search Tool is a novel web search tool that outperforms proprietary counterparts. Together with powerful open-source reasoning LLMs, such as DeepSeek-R1, ODS nearly matches and sometimes surpasses the existing state-of-the-art baselines on two benchmarks: SimpleQA and FRAMES. For example, on the FRAMES evaluation benchmark, ODS improves the best existing baseline of the recently released GPT-4o Search Preview by 9.7% in accuracy. ODS is a general framework for seamlessly augmenting any LLMs -- for example, DeepSeek-R1 that achieves 82.4% on SimpleQA and 30.1% on FRAMES -- with search and reasoning capabilities to achieve state-of-the-art performance: 88.3% on SimpleQA and 75.3% on FRAMES.

1 Introduction

Open Deep Search (ODS) is an open-source search AI intended to address the gap with proprietary search solutions. It combines an Open Search Tool with an Open Reasoning Agent and, with DeepSeek-R1, outperforms Perplexity products on both benchmarks while surpassing GPT-4o Search Preview on FRAMES.

1 Introduction

  • Search AIs combine LLM retrieval-augmented generation with real-time search-engine information to answer with up-to-date, contextually relevant responses.The introduction notes that feeding search-engine results as context to an LLM has outperformed earlier methods such as self-ask.Source: §1 · p. 1
  • ODS is introduced as an open-source search AI amid proprietary solutions that the authors say limit transparency, innovation, and entrepreneurship.The stated goal is to grow a developer community and foster collective innovation and entrepreneurship.Source: §1 · p. 1
  • ODS with DeepSeek-R1 exceeds Perplexity and Perplexity Sonar Reasoning Pro on FRAMES and SimpleQA; versus GPT-4o Search Preview, it performs better on FRAMES but slightly worse on SimpleQA.ODS-v1 uses a ReAct-based agent and ODS-v2 a CodeAct-based agent.Source: Table 1 · p. 2, §1 · p. 2
  • Open Search Tool improves on open-source alternatives by rephrasing queries when needed, extracting and reranking relevant context, and handling major sites such as Wikipedia, ArXiv, and PubMed.The authors contrast this process with alternatives that primarily pass raw search-result snippets to the LLM.Source: §1 · p. 2
  • Open Reasoning Agent interprets queries, assesses retrieved context, and selects appropriate tools, including Open Search Tool, to produce answers.The two versions use ReAct and CodeAct agents.Source: §1 · p. 2

2 Open Deep Search (ODS)

ODS pairs a user-selected base LLM with an Open Search Tool and an Open Reasoning Agent. Its search pipeline builds relevant web context, while its ReAct- and CodeAct-based agent versions use tools to answer queries.

2 Open Deep Search (ODS)

  • ODS is a plug-and-play framework combining a user-chosen base LLM with Open Search Tool and Open Reasoning Agent.The experiments use Llama3.1-70B and DeepSeek-R1 as base models.Source: §2 · pp. 2–3, Figure 1 · p. 3

2.1 Open Search Tool

  • Open Search Tool takes a query and produces relevant web passages as context for the base LLM, with query, retrieval, and augmentation stages.The authors describe its retrieval quality as critical to ODS.Source: §2.1 · p. 3

2.1.1 Query Rephrasing

  • Query rephrasing generates k context-preserving variants to bridge broad user wording and the implicit information sought.The authors report that this step improves coverage and diversity of retrieved contexts and overall system performance.Source: §2.1.1 · p. 3

2.1.2 Retrieval

  • Retrieval formats search-result snippets with metadata and prompts the LLM to prioritize reliable sources when results conflict.Metadata can include each result's title, URL, description, and date authored, if available.Source: §2.1.2 · p. 4

2.1.3 Augmentation

  • Augmentation scrapes top retrieved links, embeds passage chunks, and reranks them to add query-relevant passages to the LLM context.This step is intended to support queries requiring deeper reasoning.Source: §2.1.3 · p. 4

2.2 Open Reasoning Agent

  • Open Reasoning Agent uses the base LLM and tools, including Open Search Tool, to answer queries through either a ReAct- or CodeAct-based system.The paper names the resulting systems ODS-v1 and ODS-v2, respectively.Source: §2.2 · p. 4

2.2.1 ODS-v1 with ReAct Agent

  • ODS-v1 combines CoT consistency sampling with few-shot ReAct prompting, iterating through thought, action, and observation while using search, calculation, or continued thinking.Its dynamic few-shot selection retrieves relevant examples, and a community campaign designed 200 ReAct prompts without access to the benchmarks.Source: §2.2.1 · p. 5

2.2.2 ODS-v2 with CodeAct Agent

  • ODS-v2 adapts the search tool to SmolAgents and uses a CodeAgent, a CodeAct variation, with more advanced versions coordinating multiple tools and agents.The paper describes executable code as a way to represent and execute actions.Source: §2.2.2 · p. 6

3 Experiments

The experiments evaluate ODS on SimpleQA and FRAMES, reporting strong results on both benchmarks and examining how search use and agent components contribute to performance.

3.1 Numerical analyses on two evaluation benchmarks: FRAMES and SimpleQA

  • FRAMES contains 824 challenging multi-hop questions requiring integration of multiple Wikipedia sources, and the experiments evaluate search AIs with Internet access on it and SimpleQA.SimpleQA was designed to test factuality without browsing; FRAMES was designed to test factuality and retrieval with ground-truth Wikipedia articles provided.Source: §3.1.1 · p. 7, §3.1 · p. 7

3.1.1 Numerical analysis on FRAMES

  • ODS-v2+DeepSeek-R1 achieves 75.3% on FRAMES using 3.39 searches per query, while ODS-v1+DeepSeek-R1 achieves 56.7% with one search.ODS adapts extra searches to initial-result quality and model output; the paper contrasts this with fixed 15-search approaches and notes that harder FRAMES prompts more searches than SimpleQA.Source: §3.1.1 · p. 8, Table 2 · p. 8, Figure 5 · p. 8

3.1.2 Numerical analyses on SimpleQA

  • SimpleQA has 4,326 short-form factuality questions designed to have one indisputable answer, and the paper attributes gains to both retrieval quality and the reasoning agent.The questions span diverse topics, including science and technology, politics, and art.Source: §3.1.2 · pp. 8–9

3.1 Numerical analyses on two evaluation benchmarks: FRAMES and SimpleQA

  • 88.3% on SimpleQA and 75.3% on FRAMES are reported for ODS-v2, while ODS-v1 reaches 87.7% on SimpleQA.The paper reports that both ODS versions improve on the closed-source Perplexity solutions on SimpleQA.Source: §3.1.1 · p. 8, §3.1.2 · p. 9

3.2 Ablation study of ODS

  • On a 500-example SimpleQA subset, Llama3.1-70B rises from 21.2% to 82.4% with Search, while FRAMES drops from 34.3% to 27.6%; adding CoT-ReAct and FewShot improves both progressively.Replacing Llama3.1-70B with DeepSeek-R1 further improves ODS-v1 performance.Source: §3.2 · p. 9

4 Conclusion

ODS is presented as an open-source, plug-and-play response to the gap between proprietary search AI and open-source alternatives. With DeepSeek-R1, it reports 75.3% FRAMES accuracy, 10% above GPT-4o Search Preview.

4 Conclusion

  • The ablation study reports each component as critical to state-of-the-art performance, with SimpleQA accuracy evaluated on 500 randomly sampled examples for efficiency.Source: Table 4 · p. 11
  • ODS is designed to plug into any user-chosen base LLM, including open-source models or closed-source models accessed through APIs.The framework is described as plug-and-play.Source: §4 · p. 11
  • 75.3% accuracy on FRAMES, surpassing GPT-4o Search Preview by 10%; ODS also narrows the open-source versus closed-source gap on SimpleQA.The reported result uses DeepSeek-R1.Source: §4 · p. 12
  • Open Search Tool supplies Internet retrieval, while Open Reasoning Agent interprets tasks and calls available tools, including search, calculation, reflection, and code execution.Together with powerful reasoning LLMs, the two components are reported to achieve state-of-the-art search performance.Source: §4 · p. 12
  • The authors publicly release the implementations and invite the open-source community to build on ODS.Source: §4 · p. 12

A.1 FRAMES

The FRAMES examples illustrate how ODS combines search, reflection, and calculator tools to connect clues and answer multi-step questions, including cases where Perplexity Sonar Reasoning Pro fails or rejects a valid connection.

A.1 FRAMES

  • ODS+DeepSeek-R1 identifies the wheelbase as 112 inches (2,845 mm) and checks the conversion with Wolfram-Alpha, while Perplexity is confused by conflicting measurements.Source: §A.1 · p. 15
  • ODS+Llama3.1-70B identifies Dune: Part Two as the film, while Perplexity Sonar Reasoning Pro and ODS+DeepSeek-R1 do not determine the answer.Source: §A.1 · p. 16
  • ODS+Llama3.1-70B searches again after identifying Levi & Strauss and correctly names Dockers as its first new brand after the 1984 name change; Perplexity cannot answer from one search.Source: §A.1 · p. 17
  • ODS+Llama3.1-70B uses Wolfram to give the elapsed time as 2 years, whereas Perplexity reports 2 years and 4 months despite the question requesting years.Source: §A.1 · p. 17
  • ODS+DeepSeek-R1 continues reasoning through a Thought/Action/Observation sequence to answer the Frappuccino question correctly with 31 years; Perplexity says the context is insufficient.Source: §A.1 · p. 17
  • 31 years separate The Coffee Connection’s 1975 founding and Starbucks’ 2006 introduction of juice-blended Frappuccinos, according to ODS.The response notes that the founding year comes from external knowledge because the context lacks it.Source: §A.1 · p. 18
  • Perplexity Sonar Reasoning Pro fails on the Frappuccino question and incorrectly concludes no Japanese anti-war protester won the referenced Oscar.Its response says the clues appear contradictory, while ODS identifies Miyazaki as matching them.Source: §A.1 · pp. 18–19
  • ODS identifies Hayao Miyazaki as the Japanese filmmaker who opposed the Iraq War and won the Best Animated Feature Oscar multiple times.The response connects his anti-war stance with wins for Spirited Away and The Boy and the Heron.Source: §A.1 · p. 19

A.2 SimpleQA

The SimpleQA examples contrast ODS responses with proprietary search systems and include cases where ODS cross-checks conflicting information. They also show differing answers about Rosie Perez’s work on In Living Color.

A.2 SimpleQA

  • ODS identifies July 21, 2022, as Kaitlin Armstrong’s arraignment date, while Perplexity Sonar Reasoning Pro gives July 20.The example says ODS cross-checks the conflicting dates and selects July 21; another response in the supplied passages gives July 20.Source: §A.2 · pp. 19–20
  • ODS identifies Rosie Perez’s other job on In Living Color as segment producer after cross-checking sources.The passage contrasts this answer with Perplexity Sonar Reasoning Pro, which discusses directing segments instead.Source: §A.2 · p. 19
  • A separate response says Perez directed the Dance Bumpers segments during Seasons 2–4 (1991–1993).It describes these short sequences as transitions between sketches and commercials.Source: §A.2 · p. 20

B Question and agentic response examples for few-shot chain-of-thought to be used in the ReAct-based Open Reasoning Agent of ODS-v1

The appendix gives few-shot chain-of-thought examples for the ReAct-based Open Reasoning Agent. The examples combine web searches with calculation when a query requires comparison or arithmetic.

B Question and agentic response examples for few-shot chain-of-thought to be used in the ReAct-based Open Reasoning Agent of ODS-v1

Example 1

  • The telephone-invention example searches for dates, then calculates that the Civil War’s end was 11 years away versus 13 years for the Eiffel Tower’s completion.It concludes that the Civil War’s end was closer to 1876.Source: Example 1 · p. 21

Example 2

  • The population-density example calculates 330.7 people/km² for Japan and 419.6 people/km² for India, concluding India is denser.Source: Example 2 · p. 21

Example 3

  • The Olympic-medals example retrieves gold-medal totals and calculates that the United States has 844 more than China.The totals shown are 1,127 for the United States and 283 for China.Source: Example 3 · pp. 21–22

Example 4

  • The DNA example searches for the discoverers and the year, returning James Watson, Francis Crick, and 1953.Source: Example 4 · p. 22

Example 5

  • The building-height example retrieves both heights and calculates a 447-meter difference between the Burj Khalifa and the Empire State Building.Source: Example 5 · p. 22

Example 6

Five further examples illustrate the ReAct agent answering factual queries through web searches, with calculation used when needed. They cover space, literature, sport, invention, and history.

Example 6

  • The satellite example searches for the first country to launch a satellite and its name, answering the Soviet Union and Sputnik 1.Source: Example 6 · pp. 22–23

Example 7

  • The Big Brother example searches for the novel and publication year, identifying George Orwell’s 1984, published in 1949.Source: Example 7 · p. 23

Example 8

  • The World Cup example identifies Uruguay as host of the first FIFA World Cup in 1930.Source: Example 8 · p. 23

Example 9

  • The light-bulb example identifies Thomas Edison as the inventor and General Electric as the company he later established.Source: Example 9 · p. 23

Example 10

  • The Declaration of Independence example searches for its signing location and identifies Philadelphia’s Independence Hall.Source: Example 10 · pp. 23–24

Example 11

Examples 11–15 illustrate web-search reasoning through fact questions answered by gathering one or more pieces of information. The examples span historical, literary, geographic, and astronomical facts.

Example 11

  • The theory of general relativity was developed by Albert Einstein and published in 1915.Source: Example 11 · p. 24

Example 12

  • “To be, or not to be” is from Shakespeare’s Hamlet, and Hamlet speaks the line.Source: Example 12 · p. 24

Example 13

Example 14

  • Grover Cleveland was the first U.S. president to serve two non-consecutive terms.Source: Example 14 · p. 25

Example 15

  • Jupiter is the largest planet in the solar system, with a diameter of approximately 139,820 kilometers.Source: Example 15 · p. 25

Example 16

Examples 16–20 apply search to questions about people, places, and historical events. The final example combines a search result with a calculation of elapsed time.

Example 16

  • The Wright Flyer was the first airplane to fly, achieving its first flight in 1903.Source: Example 16 · p. 25

Example 17

Example 18

  • Novak Djokovic has won the most Grand Slam tennis titles, with 24 titles.Source: Example 18 · p. 26

Example 19

  • Queen Elizabeth II was the longest-reigning monarch in British history, with a reign of 70 years.Source: Example 19 · p. 26

Example 20

  • “As You Like It” contains “All the world’s a stage” and was first performed 421 years before 2024, in 1603.Source: Example 20 · pp. 26–27

C Related work

Related work combines retrieval with generation and reasoning, while commercial search systems and prior open-source efforts offer different approaches to web-enabled search. The section also motivates open research by citing concerns about opaque generative search.

C Related work

  • Retrieval-Augmented Generation retrieves external knowledge, while Chain-of-Thought and ReAct add reasoning steps and tool access to support factuality.Source: C Related work · p. 27
  • Commercial search systems include Perplexity, Exa, and Linkup, but opaque systems limit users’ ability to build on their advances.Source: C Related work · p. 27
  • Generative search is associated with bias [10], hallucination [26], surveillance, and dangerous information [13], motivating calls to democratize access and mitigate risks.Source: C Related work · p. 27
  • Open-source approaches include LLM-assisted access to pre-indexed scientific articles [7] and decentralized semantic search using LLM-derived embeddings [16].Source: C Related work · p. 27
  • Other work uses multiagent planning to query live web information [2] or enables multimodal interaction with live websites [34].Source: C Related work · p. 27
Loading 2503.20201v1…