PaperlayerView exact revision on arXiv

Source-linked AI summary

Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Search Agents

Shuai Wang, Haodong Chen, Yu Yin, Shengyao Zhuang, Bevan Koopman, Guido Zuccon

arXiv:2608.02751v3cs.IRcs.AIcs.CL

TL;DR

Deep-search agents typically retrieve whole webpages without using their exposed structure to constrain retrieval or access relevant sections. Sieve uses fielded Boolean candidate selection, structure-rich inspection, and selective section fetching, achieving the highest accuracy on three QA collections while using 20.7–50.6% fewer tokens than the strongest conventional Search–Visit configuration.

  • Problem

    Existing deep-search workflows retrieve whole webpages and do not let agents use webpage structure to control retrieval and content access.

  • Method

    Sieve combines fielded Boolean candidate selection, interchangeable ranking, structure-rich result inspection, and selective section fetching.

  • Results

    20.7–50.6% fewer tokens accompany Sieve’s higher accuracy than the strongest conventional Search–Visit configuration across all three QA collections.

  • Takeaways & Limitations

    The findings support preserving webpage organization as searchable and fetchable fields rather than flattening webpages for deep-search retrieval.

  • Takeaways & Limitations

    The evidence does not establish generalization to other agents, domains, or live web search, and component interactions are not fully isolated.

Abstract

from arXiv · show

Existing deep-search agents use a Search-Visit workflow that retrieves whole webpages without considering the structure they expose through titles, headings, sections, and metadata. This prevents agents from directly constraining retrieval to parts of a webpage and often carries irrelevant content into their context. We introduce Sieve, a search-inspect-fetch strategy driven by a Boolean Query Language (BQL): it searches webpage fields to filter candidates, uses an interchangeable ranker to order them, presents structure-rich result cards for inspection, and fetches only selected sections. Across three QA collections, Sieve is more accurate than the strongest conventional Search-Visit configuration on each collection while using 20.7-50.6% fewer tokens. Boolean filtering improves every tested ranker, and the accuracy-context advantage persists across retriever choices and agent backbones. Our implementation is included in the SkimSearchAgent library https://github.com/ielab/skim-search-agent.

1 Introduction

The introduction identifies a mismatch between structured webpages and whole-webpage Search–Visit workflows, motivating a search–inspect–fetch strategy that preserves structure throughout retrieval and evidence access. Sieve implements this strategy through Boolean field selection, interchangeable ranking, structure-rich inspection, and selective section fetching, improving accuracy while reducing token use.

  • Motivation: Whole-webpage visits discard actionable webpage structure, including titles, headings, sections, dates, and authors, and cannot use it to control retrieval and content access.The proposed workflow preserves webpage structure throughout the research process.
  • Method: Fielded Boolean retrieval defines eligible webpages using exact field constraints, while lexical or dense ranking orders the eligible set for inspection.This division guarantees constraint satisfaction while keeping the ranker modular.
  • Method: Sieve searches with BQL, ranks the selected webpages, exposes titles, headings, and query-focused snippets, and fetches named sections instead of complete webpages.The agent sees which sections exist before requesting one, rather than guessing a section name during search.
  • Evaluation: The study derives paired flat and structured variants of BrowseComp-Plus, HotpotQA, and MuSiQue to test webpage-structure access without changing questions or emitted text.Flat variants hide section and metadata fields, whereas structured variants make them explicitly addressable.
  • Results: Sieve achieves the highest accuracy on all three collections, with its default BM25+Dense configuration improving over the most accurate Search–Visit configuration by 1.6/3.1/0.7 accuracy points.It simultaneously uses 20.7–50.6% fewer tokens.

2 Related Work

Sieve builds on iterative retrieval for research agents, Boolean query formulation, and retrieval over webpage structure. Related work motivates fielded filtering and finer-grained access beyond conventional Search–Visit workflows.

  • Structure and retrieval granularity: Sieve combines fielded constraints, a separate ranker, structure-rich result listings, and section-level fetching.This design uses webpage structure already available in the collection.
  • Retrieval workflows for research agents: Deep-search agents commonly search for a top-k list and then visit a webpage, while planning, memory, and evidence aggregation generally retain this workflow.This literature is framed as retrieval workflows for research agents.
  • Boolean search and agent-generated queries: Boolean retrieval supports precise inclusion and exclusion through field restrictions, Boolean operators, phrases, wildcards, and ranges.These capabilities remain important in specialist search, particularly biomedical systematic reviews.
  • Structure and retrieval granularity: Focused retrieval targets passages or webpage components, while newer systems vary dense indexing granularity or construct hierarchies over unstructured text.Sieve instead uses structure already available in the collection for more granular retrieval.

3 Sieve: Search–Inspect–Fetch

Sieve preserves webpage structure throughout interaction by separating candidate eligibility, ranking, inspection, and evidence access. Its workflow combines Boolean selection, modular ranking, structure-rich result cards, and selective section fetching to control what the agent inspects and receives.

  • Structured source representation: Sieve represents webpages with metadata and ordered heading–content pairs, preserving titles, headings, section bodies, and available metadata.Search addresses indexed fields, while cards and fetch may expose additional metadata such as infoboxes.
  • Search: Boolean selection followed by ranking: BQL controls source eligibility using Boolean operators, field restrictions, phrases, wildcards, and date-range filters, while a separate ranker orders eligible webpages.The distinction lets Boolean constraints express acceptable sources while ranking determines which acceptable sources the agent inspects first.
  • Search: Boolean selection followed by ranking: If a query returns no webpages, Sieve relaxes constraints to prioritize partial matches and otherwise ranks the full collection with BM25 over positive query terms.This fallback occurs only after a zero-result query and never broadens a non-empty candidate set.
  • Inspect: structure-rich result cards: Inspection presents the top k webpages as compact cards with titles, section headings as fetch targets, and 25-token query-focused snippets.The cards provide local relevance evidence without returning the webpage body.
  • Fetch: selective section access: Fetch returns one selected named section or field rather than the concatenated webpage, enabling repeated search, inspection, and fetching within the interaction budget.Sieve’s complete strategy combines Boolean selection, modular ranking, structure-rich cards, and section-level fetching; its default uses BM25+Dense ranking.

4 Experimental Setup

The experiments use paired flat and structured versions of three QA collections to isolate access to addressable webpage structure while holding content fixed. They compare matched retrieval and agent-access conditions across rankers and backbones under fixed budgets, using collection-specific accuracy and token-use measures.

  • Collections: Paired flat and structured versions of HotpotQA, MuSiQue, and BrowseComp-Plus hold questions, webpages, and emitted text identical while making sections and metadata addressable only in the structured variant.The paired design measures the value of exposing structure as addressable fields, although headings and recoverable boundaries remain visible in flat text.
  • Collections: HotpotQA, MuSiQue, and BCP-S increase in difficulty and source complexity, with exact match used for the Wikipedia collections and LLM-judge accuracy for BCP-S.Accuracy should therefore be compared within rather than across collections.
  • Baselines: Search–Fetch is the closest no-Boolean control for Sieve, matching its rankers, top-k depth, and section fetching while omitting BQL candidate selection.Search–Visit, Search–Fetch, and Sieve use BM25, dense, and BM25+Dense ranking; one-shot and Search–AutoRead baselines provide additional retrieval and context-access comparisons.
  • Agents and retrievers: The primary agent is Tongyi-DeepResearch-30B-A3B, while the backbone study additionally evaluates Qwen-AgentWorld-35B-A3B and OpenResearcher-30B-A3B.The released ReAct scaffold is preserved while collection-access tools and condition-specific instructions vary; generations use temperature 0.6 and seed 42.
  • Budgets and metrics: Every search returns at most k=5 results, visits and fetches have 12,000-token ceilings, and iterative conditions allow at most 100 agent steps.DCI instead uses a separate one-time working set of ten retrieved webpages; Tok. measures mean total input and output tokens, and LLM calls measures mean model invocations.

5 Results

Across all three collections, Sieve improves accuracy over the strongest conventional Search–Visit configuration while using substantially fewer tokens. The results attribute this advantage to repeated search with selective section access, BQL-based candidate selection, and retrieval coupled with ranking and access.

  • Accuracy–context trade-off: 1.6, 3.1, and 0.7 points improve accuracy on HotpotQA, MuSiQue, and BCP-S, with 30.4%, 50.6%, and 20.7% fewer tokens, respectively.On BCP-S, judge accuracy and exact match both improve, by 2.5 and 3.4 points.
  • Accuracy–context trade-off: 20.7–50.6% fewer tokens accompany Sieve’s accuracy improvements over the strongest conventional Search–Visit configuration across all three collections.The default BM25+Dense setting improves accuracy on HotpotQA, MuSiQue, and BCP-S while reducing context use.
  • Mechanism: Selective section-level fetching reduces context without curtailing search, allowing continued evidence-chain following without repeatedly carrying complete webpages.Sieve makes more model calls on the Wikipedia collections and a similar number on BCP-S, so call count alone is a poor proxy.
  • Mechanism: Accuracy gains cluster on token-saving questions, while losses become more common as Sieve consumes more, aligning efficiency with effectiveness.Sieve uses fewer tokens on roughly two-thirds of BCP-S questions.
  • Mechanism: BQL improves accuracy in all nine matched Search–Fetch pairs by selecting eligible candidates, while ranking determines which candidates should be inspected first.Corpus tools still need retrieval, and ranking quality must be assessed together with downstream access rather than in isolation.
  • Mechanism: Repeated search and selective access are both necessary: one-shot baselines struggle with revision, while Search–AutoRead adds full results without comparable accuracy gains.Effective deep search therefore requires revising search as evidence accumulates while controlling what enters context.

6 Ablation and Diagnostic Analysis

Sieve’s gains arise from actionable query-focused snippets and persist across retrievers and agent backbones, with lower context use and generally higher accuracy. Diagnostic analyses show collection-specific error bottlenecks and frequent use of BQL’s field restrictions, Boolean combinations, and fallback retrieval.

  • Query-focused snippets: Query-focused snippets raise accuracy by 2.9–6.8 points across all three collections.Removing snippets significantly lowers accuracy in every case and increases model calls without recovering the loss, because snippets help agents decide which section to fetch.
  • Retriever choice: Accuracy ranges from 34.6 to 38.0 across six dense encoders, while context use remains 46k–48k tokens versus 68.1k for Search–Visit.Qwen3-Embedding-4B performs best; larger encoders tend to help but do not determine retrieval quality.
  • Agent backbone: Across three agent backbones, Sieve uses less context in eight of nine backbone–collection pairs and improves accuracy in seven.On BCP-S, all three backbones move toward higher accuracy and lower context use; with OpenResearcher, token use more than halves on both Wikipedia collections while accuracy stays within 0.6 points.
  • Where errors arise: Sieve reduces selection failures and answers more questions correctly on every collection, but the dominant bottleneck differs by collection.Retrieval failures account for 42.7–46.5% of BCP-S questions, whereas synthesis failures account for 48.2–60.1% on the Wikipedia collections; synthesis failures remain comparatively stable between systems.
  • How agents use BQL: Field restrictions appear in 42.8–69.1% of questions and Boolean combinations in 34.1–44.1%.Exact constraints and fallback retrieval are complementary: zero-hit calls trigger ranked term retrieval, which handles 36.8–53.7% of search calls.

7 Conclusion

Sieve exploits webpage structure through fielded Boolean retrieval and search–inspect–fetch, achieving the highest observed accuracy across three QA collections while using fewer tokens than conventional Search–Visit. The findings also show that preserving source organization and enabling agents to author fielded queries can make structured retrieval practical for deep-search systems.

  • 7 Conclusion: Sieve achieves the highest observed accuracy on each of three QA collections and uses 20.7–50.6% fewer tokens than the best conventional Search–Visit configuration.These gains persist across retrievers and three agent backbones.
  • 7 Conclusion: Exposing webpage structure as searchable and fetchable fields shapes what agents retrieve, inspect, and include in context.The authors argue that deep-search retrieval should preserve source organization and access information at its original granularity.
  • 7 Conclusion: Language-model agents can author and revise precise fielded Boolean queries during research while retaining explicit selection constraints.This makes capabilities long used in professional search practical for general deep-search systems.

Limitations

The evaluation has limited scope and imperfectly symmetric conditions, while several design and data factors constrain what can be attributed to Sieve. Deployment also raises risks from erroneous filtering, provenance gaps, privacy, access, and licensing concerns.

  • Scope and evaluation: The comparison uses one primary backbone and three QA collections, with transfer tests covering those collections but not establishing generalization to other agents, domains, or live web search.Incomplete judge overlays for Wikipedia conditions also make exact match the primary metric for HotpotQA and MuSiQue.
  • Experimental conditions: The experimental conditions are not fully symmetric because systems receive condition-specific manuals, some Wikipedia cells have incomplete final-answer recovery, and the early-stop rule was introduced part-way through.Appendices B.2, C.3, and C.2 document these differences rather than assuming equivalence.
  • Attribution and controls: Structured/flat controls diagnose field access rather than structure itself, while Sieve jointly changes candidate selection, result cards, and content access without fully crossing every interaction.The evidence therefore supports the complete system more directly than an independent effect of each Boolean operator; zero-hit BQL queries additionally use approximate BM25.
  • Data quality: Model-generated BCP-S section trees may contain imperfect headings or boundaries, and title-matched Wikipedia articles may have drifted from the newer snapshot because content was not checked.These conditions can affect the reliability of structure-aware inspection and fetching.
  • Ethical considerations: Fielded constraints and selective fetching can exclude relevant sources or surrounding context when metadata or generated boundaries are wrong, despite the use of released benchmarks and no user data or human-subjects research.Deployed systems should retain provenance, permit full-webpage visits, and respect access, privacy, and licensing restrictions.

A Dataset Construction and Statistics … B.3 BQL Queries and Worked Interaction

The paper constructs paired structured and flat webpage collections, then specifies Sieve’s corpus-dependent BQL grammar, executor behavior, agent instructions, and logged interaction examples. These details define how structure-aware search, inspection, and section-level fetching are implemented across the evaluated datasets.

  • A Dataset Construction and Statistics: 100,195 BrowseComp-Plus webpages and all 830 questions are retained without query-dependent corpus filtering.Every BCP-S condition searches the same complete webpage collection, preserving released identifiers, questions, and relevance annotations.
  • A Dataset Construction and Statistics: Structured and flat corpus twins expose identical identifiers, questions, qrels, and emitted text, while structured versions additionally retain fields for inspection and fetching.BM25 and dense retrieval therefore see the same content, whereas fielded retrieval can scope over sections.
  • B Method Details: The appendix covers BQL grammar, executor behavior, agent instructions, issued queries, and a complete worked interaction.The worked examples come from Tongyi-DeepResearch-30B-A3B logs on BCP-S, HotpotQA-S, and MuSiQue-S.
  • B.1 Query Grammar and Tool Behavior: BQL supports field restrictions, Boolean operators, grouping, wildcards, exact phrases, and typed date ranges.Available fields vary: Wikipedia indexes title, section, and body, while BCP-S also indexes author and date; infoboxes support inspection and fetching.
  • B.1 Query Grammar and Tool Behavior: When hard AND queries return no exact matches, the default executor ranks webpages using positive query terms rather than enforcing strict Boolean retrieval.This fallback was enabled in every primary experiment; an appendix control disables it while keeping the Sieve workflow fixed.
  • B.2.1 Instruction Assignment by Condition: Query-language manuals are assigned only to conditions that expose query languages, including Sieve rankers and the appendix-only Indri comparison.BM25, dense, BM25+Dense, DCI, Search–Visit, Search–Fetch, and retrievethen-read receive no query-language manual.
  • B.2 Query-Language Instructions: Sieve receives identical Search–Fetch instructions on HotpotQA and MuSiQue, while BCP-S instructions retain the workflow but add corpus-specific author and date guidance.The BCP-S manual advises falling back to body text when an author is unavailable; Indri receives a separate, collection-independent syntax manual.
  • B.3 BQL Queries and Worked Interaction: The logged BQL examples and complete interaction instantiate Sieve’s search, inspection, and fetching workflow across BCP-S, HotpotQA-S, and MuSiQue-S.Instance identifiers are abbreviated by dataset prefix in the appendix examples.

B.3.1 Sample of Issued Queries … C.2 Decoding, Serving, and Budgets

The appendix illustrates Sieve’s issued-query grammar and a complete search–inspect–fetch interaction, then documents experimental baselines, DCI deviations, and shared serving and budget settings. These details establish how candidates were queried, inspected, fetched, evaluated, and compared across conditions.

  • B.3.1 Sample of Issued Queries: Table 5 lists 12 agent-issued queries covering the grammar’s main constructs; zero exact matches trigger a soft BM25-ranked fallback.“0 (fallback)” advises the agent to verify, loosen, or replace the query.
  • B.3.2 A Worked Interaction: The worked interaction searches for Fort Santiago across title and body fields, returning five matches and structure-rich result cards.The query compiles to IN(title, "Fort Santiago") OR IN(body, "Fort Santiago").
  • B.3.2 A Worked Interaction: After inspecting headings, the agent fetches selected sections whose text supplies evidence for the question’s historical constraints.The fetched passages describe the 1574 Limahong attack, the 1590–1593 rebuild, and later earthquake damage.
  • B.3.2 A Worked Interaction: The model checks all four constraints against fetched text and outputs Fort Santiago, receiving exact-match credit.The constraints cover 16th-century construction, Manila’s historical context, the attack three years later, same-century rebuilding, and mid-17th-century damage.
  • C Experimental Details and Baselines: The appendix reports baseline implementation details, common serving and budget settings, evaluation procedures, and retrieval-engine validation.It states that the experimental setup follows the main paper.
  • C.1 DCI Reimplementation: DCI searches the raw corpus using only bash and read tools, while the BM25-bounded control stages ten webpages before shell-based access.The BM25-bounded condition uses the Search–Visit BM25 implementation and enforces the bound through the staged filesystem.
  • C.1 DCI Reimplementation: DCI omits the original five-step strategy prompt and receives at most 100 steps, potentially understating performance relative to its published implementation.The original paper used 300 turns and reported that DCI needs more tool calls than a retrieval agent to find an initial anchor webpage; query-language conditions also receive a tool manual.
  • C.2 Decoding, Serving, and Budgets: All backbones use temperature 0.6, seed 42, local vLLM serving, and fixed ReAct scaffolding; iterative runs return at most k=5 cards and allow 100 steps.The primary context window is 131,072 tokens, while Qwen-AgentWorld and OpenResearcher use 262,144; runs may stop at 90% context usage or after answering.

C.3 Evaluation and Answer Recovery … D.2 Efficiency–Effectiveness Comparison

The evaluation controls answer recovery, budget matching, and retrieval-engine validation before presenting complete results and efficiency–effectiveness analyses. These analyses document recovery asymmetries, fallback-ranking divergence, structured-executor comparisons, expanded collection results, and the BCP-S accuracy–context trade-off.

  • C.3 Evaluation and Answer Recovery: Recovery overlays are applied before judged comparisons because empty-run frequency varies and recovered answers can change system rankings.A deterministic terminal-transcript continuation recovers about 93% of applicable runs without further tool use.
  • C.3 Evaluation and Answer Recovery: Recovery coverage is incomplete: all BCP-S cells and two Wikipedia Search–Visit baselines have overlays, but Wikipedia Sieve and other Search–Fetch conditions do not.This asymmetry limits comparisons across those groups.
  • C.3 Evaluation and Answer Recovery: Agentic comparisons hold the per-webpage ceiling at 12,000 tokens, the maximum at 100 agent steps, and search results at five per call.Whole-webpage visit versus section fetch therefore changes returned content rather than the maximum available from a webpage; prompt-fitting also enforces retrieve-then-read budgets after full tokenization.
  • C.4 Retrieval Engine Controls: Zero-hit Sieve queries use partial-match coverage ranking first and an approximate BM25 fallback over positive terms when no partial match exists.The fallback applies directly to zero-hit single-constraint queries.
  • C.4 Retrieval Engine Controls: 36.8–53.7% of Sieve search calls use zero-hit fallbacks, whose approximate BM25 top-5 Jaccard overlap with Lucene BM25 is 0.546 on BCP-S.Reported results therefore combine Lucene Boolean filtering and ordinary sparse ranking with approximate BM25 only in the fallback path, rather than constituting a pure Lucene evaluation.
  • C.4 Retrieval Engine Controls: An Indri-style executor is reported separately as an established structured-retrieval alternative, while holding result cards, section-fetch behavior, result-pool size, and content-access budget fixed.Its condition is excluded from the main system comparison.
  • D Full Results and Additional Analyses: The appendix presents complete system results, efficiency–effectiveness comparisons, component and robustness analyses, and failure diagnostics in the order of the main findings.Expanded tables cover HotpotQA, MuSiQue, and BCP-S; recall, token, accumulated-token, and model-call measures are defined for these results.
  • D.2 Efficiency–Effectiveness Comparison: Figure 8 places primary-backbone BCP-S configurations, including retriever-sensitivity and structured-executor controls, on a common judge-accuracy versus distinct-token plane.The figure marks BM25 Search–Visit with a square, Sieve with a star, and connects them with an arrow.

D.3 Component and Robustness Analyses · D.4 Hand-Traced Failure Examples

The component and robustness analyses evaluate snippet inclusion, retriever and agent-backbone sensitivity, BQL usage, fallback controls, and structured-corpus effects. Hand-traced examples distinguish retrieval, selection, and synthesis failures in five incorrect Sieve runs.

  • D.3 Component and Robustness Analyses: Matched snippet ablation compares result cards with and without a 25-token query-focused snippet using accuracy and cost measurements.Accuracy uses judge verdict on BCP-S and exact match on HotpotQA and MuSiQue; cost follows §4.4.
  • D.3 Component and Robustness Analyses: Dense-encoder sensitivity tests vary only the retriever across BCP-S configurations, with the default encoder identified explicitly.Table 11 reports the complete accuracy and cost measurements summarized in Figure 5.
  • D.3 Component and Robustness Analyses: Agent-backbone transfer pairs BM25 Search–Visit with Sieve for each backbone and collection under the same serving configuration.Accuracy is the judge verdict on BCP-S and exact match elsewhere; token counts are distinct tokens per question.
  • D.3 Component and Robustness Analyses: BQL-use analysis reports how often each Boolean Query Language operator class appears at least once in a question.Operator-class rows overlap, and the no-field/Boolean/date category may still include phrases or wildcards.
  • D.3 Component and Robustness Analyses: Additional controls remove ranked fallback and compare Search–Visit over structured versus flat corpus twins.Table 14 also controls snippets and field exposure on BCP-S.
  • D.4 Hand-Traced Failure Examples: Five incorrect Sieve runs are manually examined as diagnostic examples rather than a representative sample.The traces assess the boundaries used by the automatic failure decomposition in Section 6.2.
  • D.4 Hand-Traced Failure Examples: Three runs are retrieval failures because no gold webpage identifier appears in any search observation, despite incorrect answers for their questions.The gold answers are Boston, Laura Lojo-Rodriguez, and Georgia Hirst; the model answers Santiago, Muhammad Faruque, and 1845.
  • D.4 Hand-Traced Failure Examples: The remaining traces show selection and synthesis failures: one surfaced necessary evidence but fetched unrelated sections, while another fetched answer-containing evidence but reasoned to the wrong answer.For __788, the gold answer is Taj-ul-Masajid and the model returns Faisal Mosque; for __773, the gold answer is Red and the model answers white.
Loading 2608.02751v3…