PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?

Tz-Huan Hsu, Jheng-Hong Yang, Jimmy Lin

arXiv:2605.10848v1cs.IRcs.AIcs.CL

TL;DR

As LLMs become stronger in agentic loops, the paper asks whether lexical retrieval remains sufficient for deep research. It introduces PI-SERINI, which combines a tuned BM25 retriever and deeper retrieval with tools for searching, browsing, and reading documents, achieving 83.1% answer accuracy and 94.7% surfaced evidence recall on BrowseComp-Plus.

  • Problem

    The paper revisits whether lexical retrieval remains sufficient for deep research as LLMs gain stronger reasoning and tool-use abilities in agentic loops.

  • Method

    PI-SERINI pairs tuned BM25 and sufficient retrieval depth with a minimal search agent providing tools to search, browse, and read documents.

  • Results

    83.1% answer accuracy and 94.7% surfaced evidence recall were achieved on BrowseComp-Plus, outperforming released dense-retriever search agents.

  • Takeaways & Limitations

    On BrowseComp-Plus, well-configured BM25 with sufficient retrieval depth can support effective deep research with capable LLMs in an agentic loop.

  • Takeaways & Limitations

    The evaluation is limited to BrowseComp-Plus, leaving generalization to multilingual queries and domain-specific scenarios open.

Abstract

from arXiv · show

Does a lexical retriever suffice as large language models (LLMs) become more capable in an agentic loop? This question naturally arises when building deep research systems. We revisit it by pairing BM25 with frontier LLMs that have better reasoning and tool-use abilities. To support researchers asking the same question, we introduce Pi-Serini, a search agent equipped with three tools for retrieving, browsing, and reading documents. Our results show that, on BrowseComp-Plus, a well-configured lexical retriever with sufficient retrieval depth can support effective deep research when paired with more capable LLMs. Specifically, Pi-Serini with gpt-5.5 achieves 83.1% answer accuracy and 94.7% surfaced evidence recall, outperforming released search agents that use dense retrievers. Controlled ablations further show that BM25 tuning improves answer accuracy by 18.0% and surfaced evidence recall by 11.1% over the default BM25 setting, while increasing retrieval depth further improves surfaced evidence recall by 25.3% over the shallow-retrieval setting. Source code is available at https://github.com/justram/pi-serini.

1 Introduction

The introduction revisits whether lexical retrieval remains sufficient for deep research as LLM reasoning and tool use improve. It presents Pi-Serini, a minimal tool-using agent, and reports that well-configured BM25 at sufficient depth can support strong BrowseComp-Plus performance.

  • Agent design: Pi-Serini provides search, result browsing, and document reading as distinct decisions for selective evidence acquisition.The agent caches retrieved rankings and lets the LLM explore deeper results and choose which evidence enters its context window.
  • Results: 83.1% answer accuracy and 94.7% surfaced evidence recall are achieved by Pi-Serini with gpt-5.5 on BrowseComp-Plus under time-budget steering.This outperforms released search agents based on dense retrievers.
  • Ablations: 18.0% higher answer accuracy and 11.1% higher surfaced evidence recall result from BM25 tuning over the default BM25 setting.Increasing retrieval depth further improves surfaced evidence recall by 25.3% over the shallow-retrieval setting.
  • Motivation: BM25 can support effective deep research when paired with capable LLMs, sufficient retrieval depth, and an interface for managing cached rankings.The experiments separate lexical retrieval capacity from configuration and retrieval depth.
  • Efficiency: 3.3×–10× lower evaluation cost is reported for BM25-based search agents, while prefix caching, BM25 tuning, and wall-clock budgeting improve cost efficiency.Figure 1 summarizes the resulting accuracy–cost trade-off, with Pi-Serini consistently in the efficient region.

2 Related Work

Deep research extends RAG with iterative search, document inspection, and multi-source evidence synthesis. Prior work has explored retrievers, reranking, and document ordering, but has not isolated whether BM25’s weakness stems from lexical retrieval or surrounding design choices.

  • Deep Research: Deep research agents extend single-step RAG with iterative search, document inspection, and evidence synthesis across multiple sources.These multi-step retrieval-and-reasoning processes address complex information-seeking tasks.
  • Retriever for Deep Research: Search agents use dense retrievers, sparse retrievers, and rerankers to improve early-stage ranking accuracy.Reasoning-aware retrievers additionally target implicit intent and multi-hop information needs, while BM25 is commonly treated as a non-dense baseline.
  • Related Analyses: Prior analyses study LLM-based retrieval, reranking, and document ordering in search agents for deep research.These studies examine retrieval effectiveness and how ranking choices affect agent performance.
  • Related Analyses: Existing studies do not isolate whether BM25’s apparent weakness comes from lexical retrieval itself or from configuration and retrieval depth.This gap motivates examining BM25 independently from surrounding design choices.

3 PI-SERINI

Pi-Serini isolates agent–retriever interaction by placing a retrieval controller between a ReAct-style LLM agent and an Anserini BM25 backend. Its separate retrieval, browsing, and reading tools support selective evidence access, trajectory logging, and time-budgeted completion.

  • Architecture: Pi-Serini runs an LLM agent in a ReAct loop, with a retrieval controller mediating all access to the Anserini BM25 backend.The controller exposes a constrained tool API, caches rankings, and controls how retrieved evidence enters the agent context.
  • Agentic Loop: The agentic loop models each turn as reasoning, an action, and an observation, with the final action producing the response and terminating the loop.Intermediate actions may be tool calls or reasoning, and the observed documents form the retrieval-process document set.
  • Tool Interface: Its three search tools separate lexical retrieval, cached-result browsing, and line-based document reading for selective evidence access.Search retrieves up to 1000 ranked documents but initially exposes only the top 5 excerpts; browsing paginates cached rankings, while reading fetches relevant document text.
  • Trajectory Logging: Pi-Serini logs four document sets—Dsurfaced, Dpreviewed, Dopened, and Dcited—to distinguish available, inspected, read, and cited evidence.This trajectory artifact records the full tool-call and reasoning trajectory for each query.

4 Experimental Setup

The experiments evaluate PI-SERINI with dense-retriever and BM25 pairings across LLMs on BrowseComp-Plus, measuring answer quality and retrieval effectiveness. PI-SERINI uses tuned BM25 retrieval, frontier LLMs, and a custom deep-research agent harness.

  • Baselines: The study compares search agents pairing different LLMs with dense retrievers or BM25 for retrieval and final-answer quality.Released baselines use o3 or gpt-5 with qwen3-embed-8b or BM25; additional comparisons include gpt-5.2 with qwen3-embed-8b and AgentIR.
  • Dataset: PI-SERINI is evaluated on BrowseComp-Plus, comprising 830 queries and 100,195 documents.Each query averages 6.1 evidence documents and 2.9 gold documents.
  • Metrics: Answer evaluation reports Accuracy and Calibration Error, while retrieval evaluation reports Surfaced Recall, Previewed Recall, and Behavior Recall.An LLM judge running with gpt-5.3-codex determines whether each extracted final answer is semantically equivalent to the benchmark answer.
  • Retriever: PI-SERINI indexes BrowseComp-Plus with ANSERINI BM25 using k1 = 25, b = 1, and retrieval depth 1000 across all runs.The parameters are tuned for long-document retrieval in a deep-research setting.
  • Models: The evaluation spans frontier LLMs from DeepSeek, Anthropic, and OpenAI to test PI-SERINI with more capable reasoning and tool-use models.The models include deepseek-v4-flash, deepseek-v4-pro, claude-haiku-4.5, claude-opus-4.7, and multiple GPT-5 variants through gpt-5.5.
  • Agent Harness: PI-SERINI uses a custom PI-based agent harness that instructs agents to issue queries, inspect retrieval results, and synthesize answers within T = 300 seconds per query.The harness removes PI’s system prompts and built-in tools and replaces them with purpose-built search tools.

5 Main Experimental Results

PI-SERINI shows that a well-configured BM25 retriever with sufficient retrieval depth can support competitive deep research when paired with capable LLMs. It achieves strong answer quality and surfaced recall while reducing cost and, in comparable settings, using fewer tool calls than released baselines.

  • Answer Quality: 16.3 percentage points higher accuracy: PI-SERINI improves the released gpt-5 + BM25 baseline from 58.3% to 74.6%.Under the same LLM and BM25 setting, it also slightly exceeds the released gpt-5 + qwen3-embed-8b dense-retriever baseline.
  • Answer Quality: 83.13% accuracy makes PI-SERINI with gpt-5.5 the highest-accuracy BM25 system and the highest-accuracy system in Table 1.The result is attributed to well-configured BM25, sufficient retrieval depth, and a more capable LLM in the agentic loop.
  • Retrieval Behavior: 94.7 evidence-document recall and 94.4 gold-document recall: gpt-5.5 + BM25 substantially exceeds dense-retriever baselines at 79.21 and 81.34, respectively.Most PI-SERINI runs achieve over 90% surfaced recall on both evidence and gold documents.
  • Cost Analysis: $196.7 lower cost: under the same gpt-5 + BM25 setting, PI-SERINI reduces cost from $400.4 to $94.9.With gpt-5.5, it achieves the table’s highest accuracy at $291.6, below the released gpt-5 baselines.
  • Tool Usage: 8.0 fewer tool calls per query: PI-SERINI uses 15.2 under gpt-5 + BM25, compared with 23.2 for the released baseline.Most PI-SERINI systems use a similar or smaller number of total tool calls than released baselines, except the claude-haiku-4.5-based agent.

6 Ablation Study

The ablations show that retrieval depth and BM25 retuning substantially improve surfaced evidence recall and answer accuracy, while termination policy and prefix caching shape cost efficiency. Larger retrieval depth raises surfaced recall, but previewed recall saturates, and cached-token ratios remain high across settings.

  • Retrieval Depth: 95.8% surfaced recall is reached at k = 1000, up from 70.5% at k = 5.Increasing k from 5 to 100 raises surfaced recall from 70.5% to 86.22%.
  • Retrieval Depth: Previewed recall peaks at 74.7% with k = 50 and does not keep improving as retrieval depth increases.Larger cached rankings place more relevant evidence in the ranking but do not automatically lead the agent to inspect more of it.
  • Tuning BM25 for Long Documents: k1 = 16 and b = 1.0 is near the best BM25 configuration, whereas the default k1 = 0.9 and b = 0.4 lies in a low-performing region.The default setting is underfit for long-document evidence search.
  • Tuning BM25 for Long Documents: 18.0 percentage points: tuning BM25 raises accuracy from 64.0% to 82.0% and surfaced recall from 84.6% to 95.7%.The tuned setting uses k1 = 25 and b = 1; the best grid-search setting yields 81.1% accuracy and 94.0% surfaced recall.
  • Cost-Efficiency: $24.9 to $21.7: replacing the 100-iteration cap with a 300-second timeout slightly reduces cost, while 82% to 90% of tokens are served from cache across settings.The high cached-token ratios suggest that prefix-cache-friendly interaction contributes substantially to overall cost efficiency.

7 Discussion

The discussion contrasts frontier-model behavior and retrieval formulations, noting a performance gap despite comparable inference costs and arguing that properly configured BM25 can provide high-recall candidates without pre-localizing the corpus.

  • Failure Mode Analysis: Frontier models with comparable inference costs nevertheless show a substantial performance gap on BrowseComp-Plus.The comparison concerns GPT-5.5 and Claude Opus 4.7 in the authors’ setting.
  • Failure Mode Analysis: Both agents sometimes expand queries using internal knowledge or weakly related entities from retrieved documents.The trajectory analysis identifies these behaviors as occurring during search.
  • To Retrieve or Not To Retrieve: Coding-agent approaches reformulate deep research as local file-system navigation over a materialized document subset.They use command-line tools such as rg, sed, and nl to identify needed context through simple interactions.
  • To Retrieve or Not To Retrieve: Local navigation assumes the relevant search space can be localized in advance, which is difficult for large, dynamic real-world corpora.This assumption limits how directly the formulation transfers beyond settings where relevant documents are already locally available.
  • To Retrieve or Not To Retrieve: With proper configuration and sufficient retrieval depth, BM25 can provide a high-recall candidate set for the agent to inspect.This is presented as complementary to local-navigation systems whose relevant documents are already available.

8 Conclusion

On BrowseComp-Plus, a well-configured lexical retriever with sufficient retrieval depth can support effective deep research when paired with more capable LLMs in an agentic loop. Pi-Serini’s key limitation is that the agent does not always effectively use evidence it has already surfaced.

  • Conclusion: A well-configured lexical retriever with sufficient retrieval depth can suffice for effective deep research with more capable LLMs in an agentic loop.On BrowseComp-Plus, it can match or exceed released dense-retriever baselines.
  • Conclusion: The prefix-cache-friendly agent harness makes full-benchmark runs far cheaper.
  • Limitations: Pi-Serini often surfaces the right evidence documents, but its agent does not always browse, open, or use them effectively.This shifts the improvement question from whether retrieval finds evidence to whether the agent recognizes and spends context on evidence already provided.

9 Limitations · A System Setup

PI-SERINI’s limitations concern its minimal cached-ranking navigation interface and evaluation scope, while the appendix specifies the agent setup by prompt, tools, backend, and runtime policy. The documented runtime policy includes steering the time budget toward answering.

  • 9 Limitations: PI-SERINI’s minimal interface for navigating cached rankings yields substantially lower previewed and behavior recall than surfaced recall.
  • 9 Limitations: The evaluation is limited to BrowseComp-Plus, leaving generalization of the current agent harness unresolved.
  • A System Setup: The appendix documents the exact PI-SERINI agent setup used in the experiments.
  • A System Setup: The system setup is organized by prompt, tool interface, backend configuration, and runtime policy.
  • A System Setup: Curly-braced placeholders such as {Question} are replaced at runtime.
  • A System Setup: The runtime policy includes steering the time budget toward answering.

Appendix A.1 retrieval workflow … A.5 Gold-Answer LLM Judge

PI-SERINI separates the LLM agent from ANSERINI BM25 through a retrieval controller exposing search, browsing, and reading tools with managed retrieval state. The appendix also specifies the BM25 backend, time-budget steering, and a gold-answer LLM judge for final-answer correctness.

  • A.1 Agent Prompt: The agent uses a fixed final-response format and is guided by a retrieval-workflow prompt and time-budget steering policy.The documented architecture identifies these components as part of PI-SERINI.
  • Appendix A.1 retrieval workflow: The retrieval controller isolates the agent from ANSERINI and exposes constrained search, browse, and read tools.It maintains local retrieval state for controlled, paginated BM25 retrieval.
  • A.2 Retrieval Tool Interface: Search submits a plain raw query, requests up to 1000 hits, and initially displays ranks 1–5; browsing then paginates the cached ranking.read_search_results defaults to offset 6 and limit 10 when omitted.
  • A.2 Retrieval Tool Interface: Reading retrieves one backend document by docid in line-based chunks, defaulting to offset 1 and at most 200 lines.The output includes line ranges, total line count, truncation metadata, and a next offset when available.
  • A.3 Backend and Cache Configuration: The experimental backend is Anserini BM25 on BrowseComp-Plus with k1 25 and b 1.Search results are cached by session-local search_id, with at most 32 entries retained and older entries evicted first.
  • A.4 Time-Budget Policy: At T = 300 seconds, the submit-now policy steers the agent to stop using tools and submit its best answer when the timer fires.Later calls to search, read_search_results, and read_document are blocked, while the final format remains “Explanation: ... Exact Answer: ... Confidence: ...”.
  • A.5 Gold-Answer LLM Judge: Final-answer correctness is judged by an isolated openai-codex/gpt-5.3-codex call receiving the question, agent response, and benchmark correct answer.Each judge call runs in JSON mode and times out after 180 seconds.
  • A.5 Gold-Answer LLM Judge: The judge compares semantic equivalence without solving the question or using outside knowledge, allowing harmless wording differences and equivalent numerical forms.It returns exactly one JSON object containing the extracted answer, correct answer, reasoning, boolean correctness, and confidence from 0 to 100.

B Model Token Pricing Used in Benchmark Runs

This section reports the token pricing used during benchmark execution, with all prices expressed in USD per 1M tokens.

  • Table 4 reports the token pricing used by the benchmark runtime.
  • All prices are denominated in USD per 1M tokens.
  • The reported pricing applies during benchmark execution.

C Premature Branch Commitment

The section contrasts reversible exploration with premature branch commitment in agentic search. GPT-5.5 recovers from unverified query expansions, whereas Claude Opus 4.7 deepens an initially superficial candidate anchor without testing alternatives.

  • GPT-5.5: GPT-5.5 introduces unverified terms, including Warrington, Vinegar Strokes, and Arts Educational, but abandons unproductive directions and returns to root constraints.It then follows evidence-supported “Crawley” probes using the town-population and spelling-history clues.
  • Claude Opus 4.7: Claude Opus 4.7 anchors on a superficial Renato Paris and CICAS title match, then searches vertically within that branch.The branch adds unverified terms such as Tom Misch, Italy, Forlì, ICMP, BIMM, and Berklee instead of testing alternative interpretations.
  • Task constraints: The original task combines interview quotations, a town population near 114,000, a spelling-history clue, career milestones, and a school-identification target.These constraints provide the root conditions for evaluating whether query trajectories remain evidence-supported.
  • Evidence-supported recovery: The successful trajectory identifies Crawley from the population and spelling clues, then searches for an artist matching the interview and career constraints.A later probe names Dani Sylvia as matching Crawley, a 2019 release, and a songwriter award while seeking the school answer.
Loading 2605.10848v1…