PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Search in Real-World Web Environments

Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin

arXiv:2610.30181v1cs.IRcs.MMcs.SE

TL;DR

Real-world Web research requires agents to gather and integrate evidence across diverse modalities, formats, and interactive sources, yet existing evaluations do not adequately test these demands. This paper defines OpenDS and builds OpenDS-World, a 200-task benchmark for multi-step evidence search, then evaluates 12 frontier models. The strongest model reaches 39.0% accuracy, while analyses reveal persistent challenges in agent research strategies and execution.

  • Problem

    Existing Deep Search evaluations do not sufficiently assess the interconnected evidence-acquisition demands of real-world Web environments, where evidence spans diverse modalities and formats and may require interactive access.

    Source: §1 · p. 3

  • Method

    The paper defines OpenDS as multi-step search, access, processing, and integration of evidence across heterogeneous Web sources, and constructs a 200-task benchmark from audiovisual anchors and independently verified evidence paths.

    Source: §2.1.1 · p. 5, §2.2.1 · p. 6

  • Results

    39.00% accuracy is achieved by GPT-6 Astra, the highest overall result on OpenDS-World; human participants achieve 21.1% across independent attempts, and analyses find substantial differences in model search strategies and execution behavior.

    Source: §1 · p. 4, §3.2 · p. 11

  • Takeaways & Limitations

    OpenDS-World provides a unified platform for systematically evaluating agents’ deep-search ability across heterogeneous information sources in real-world Web environments.

    Source: §1 · p. 4

Abstract

from arXiv · show

Despite substantial progress in Deep Search agents, their ability to conduct end-to-end research in real-world open Web environments remains insufficiently evaluated. Unlike static retrieval settings, information on the real-world Web is distributed across diverse sources, modalities, and heterogeneous formats, often involving complex interaction mechanisms that require agents to autonomously search for, access, and integrate relevant evidence during execution. To address this gap, we introduce the OpenDS task, designed to evaluate agents' ability to perform multi-step deep search in the live, open Web. This task requires agents to autonomously explore the real Web, discover and connect relevant evidence from multiple independent sources through Web search, browser interaction, and evidence processing, and ultimately determine a unique factual answer. Building on this task formulation, we construct OpenDS-World, a benchmark comprising 200 challenging tasks spanning diverse information modalities, heterogeneous file formats, and interactive access requirements, with independent human verification to ensure evidence validity and answer uniqueness. We evaluate 12 frontier models under a unified tool environment and execution protocol. The best-performing model achieves 39.0% accuracy, while human evaluation across 1,000 independent attempts yields 21.1% accuracy. Further analyses of tool usage, search cost, termination behavior, and failure recovery reveal substantial differences in research strategies across models, as well as persistent challenges in tool coordination and long-horizon search management. OpenDS-World provides a unified evaluation platform for systematically assessing and advancing agents' autonomous deep search capabilities in real-world open Web environments.

1. Introduction

Existing evaluations do not adequately capture the interconnected evidence-acquisition demands of real-world Web research. OpenDS and its 200-task benchmark target those demands, and evaluation shows that frontier agents still struggle with them.

1. Introduction

  • Existing benchmarks insufficiently cover the interconnected demands of diverse modalities, heterogeneous formats, and interactive access that characterize real-world Web research.Agents must coordinate discovery, access, resource processing, and cross-source reasoning over multiple steps.Source: §1 · p. 3
  • 39.0% accuracy is achieved by GPT-6 Astra, the best-performing model; human participants achieve 21.1% across independent attempts, with 42.5% of tasks unsolved by all assigned participants.Analyses also find model differences in tool use, search cost, termination, and failure recovery; more tool calls and longer execution do not necessarily yield higher accuracy.Source: §1 · p. 4
  • OpenDS defines a multi-step task in which agents search, access, process, and connect evidence across heterogeneous Web sources to determine a unique factual answer.Tasks in OpenDS-World start from a video and require evidence discovery across multiple sources; an example connects audiovisual identification, virtual-tour navigation, document inspection, and structured-data retrieval.Source: §1 · pp. 3–4, Figure 2 · p. 4
  • OpenDS-World comprises 200 tasks across eight domains and covers diverse modalities, heterogeneous formats, and seven access types.Independent human verification and filtering are used to ensure evidence validity, factual accuracy, and answer uniqueness.Source: §1 · p. 4

2. OpenDS-World

OpenDS-World operationalizes deep search as multi-step evidence gathering and reasoning across real-world Web sources. The section defines the task and its challenges, describes benchmark construction and validation, and reports human-solving difficulty and task diversity.

2. OpenDS-World

  • OpenDS-World contains 200 challenging tasks built from real-world Web sources across eight domains, evaluating discovery and connection of evidence across modalities, formats, and interactive access requirements.The benchmark focuses on evidence acquisition in complex real-world Web environments.Source: §2 · p. 5

2.1.1. Task Definition

  • OpenDS is factual question answering that requires multi-step search and reasoning across heterogeneous sources, whose evidence locations and access methods are not given in advance.Agents must adapt search targets and access or processing methods as clues accumulate, then connect evidence across sources to determine a unique, verifiable answer.Source: §2.1.1 · p. 5

2.1.2. Main Attributes

  • The task spans diverse modalities, heterogeneous formats, and interactive access, which shape how agents discover, acquire, and connect evidence.Sources may require cross-modal interpretation, format-specific processing, or operations such as navigation, parameter input, temporal indexing, graphical manipulation, structured queries, and downloads.Source: §2.1.2 · p. 6

2.2.1. Task Construction

  • Task construction starts from an audiovisual entity anchor, recursively expands a multi-source evidence path, and inverts that path into interdependent clues that uniquely identify the terminal answer.Nodes are independently corroborated, and the answer cannot be retrieved from a single explicit clue.Source: §2.2.1 · pp. 6–7

2.2.2. Human Verification

  • Ten independent reviewers check evidence-path validity and task validity, including factual support, question clarity, constraint consistency, and answer uniqueness.Candidates with errors, insufficient evidence, ambiguity, non-unique answers, or construction violations are excluded; the final benchmark has 200 tasks with defined answers and verifiable paths.Source: §2.2.2 · p. 7

2.3. Human Testing

  • Human testing measures the practical difficulty of independently finding evidence paths and solving the questions in a real Web environment.The evaluation provides human performance data for difficulty calibration.Source: §2.3 · p. 7

2.3.1. Independent Human Solving

  • Solvers worked independently in groups of five on disjoint sets, could use conventional browsers but not LLM tools, and submitted answers, solving times, and difficulty ratings.No uniform per-question time limit was imposed, and final answers were manually checked.Source: §2.3.1 · pp. 7–8

2.3.2. Human Performance Analysis

  • 21.10% of 1,000 human attempts were correct, and 42.50% of tasks were unsolved by all five solvers; the 50 participants spent about 646 hours in total, averaging 12.92 hours per 20-task set.The mean subjective difficulty rating was 3.884/5, and 65.50% of attempts were rated 4 or 5.Source: §2.3.2 · p. 8

3. Experiments

Across the unified evaluation, frontier models achieve limited accuracy, while search costs, tool-use patterns, and execution behavior vary; budget exhaustion and tool failures remain challenges.

3.1. Setting

  • The 12 models use the same inputs, prompts, tools, and execution budgets; search ends at 3,600 seconds or 100 calls, followed by evidence-only finalization if needed.A Qwen3.8-27B judge evaluates semantic answer equivalence; accuracy is primary, with token consumption and elapsed time as resource-cost metrics.Source: §3.1 · p. 10

3.2. Main Results

  • GPT-6 Astra achieves the highest overall accuracy, 39.00% across 200 tasks, but fails on 122 tasks (61.00%).Higher resource use does not consistently correspond to higher accuracy: GPT-6 Astra and GPT-5.6 Sol consume nearly identical total tokens but differ by 11.00 percentage points.Source: §3.2 · p. 11
  • Mean accuracy across the 12 models falls from 39.66% on Low tasks to 23.50% on Medium and 16.76% on High.Rankings vary by difficulty: GPT-6 Astra leads Low and ties with Claude Opus 5.5 on Medium, while Gemini 3.8 Flash leads High at 28.24%.Source: §3.2 · p. 11

3.3. Cost Analysis.

  • GPT-6 Astra averages 51.05 tool calls and 18.66 minutes per task; tool use rises from Low to Medium for all models but varies from Medium to High.Search cost does not consistently track accuracy; task-level runtimes also show pronounced long tails for several models.Source: §3.3 · pp. 11–12, Figure 6 · p. 11, §3.2 · p. 11

3.4. Tool-Use Analysis

  • GPT-5.6 Sol devotes 74.50% of calls to Search, while similar GUI shares for Claude Opus 5.5 and Qwen3.8-Flash accompany 38.00% versus 12.00% accuracy.Within-category transitions dominate for some models, including Search→Search at 84.63% for Gemini 3.8 Flash and GUI→GUI at 84.97% for Qwen3.8-Flash.Source: §3.4 · p. 12, Figure 7 · p. 12

3.5. Error Analysis

  • Tool failures account for 2.83%–9.42% of calls but consume 19.27%–61.90% of execution time; switching tools is the most common post-failure decision.For GPT-6 Astra, failed calls are 6.34% of calls and consume 42.57% of tool time.Source: §3.5 · pp. 12–13, Figure 8 · p. 13

3.6. Budget Analysis

  • At least one execution limit is reached on 81.5% of Qwen3.8-Flash tasks, mostly the call limit; at most 1.0% of tasks per model exhaust both limits.Several models are predominantly time-limited rather than call-limited, including GLM 5.3, which reaches the time limit alone on 41.0% of tasks.Source: §3.6 · p. 13, Figure 9 · p. 13

4. Related Work

Related work examines open-Web deep search, multimodal evidence grounding, and interactive access to Web sources.

Deep search in open Web environments

  • Open-Web deep search work spans retrieval and reasoning, adaptive search, trajectory coordination and state management, and benchmarks for evidence depth, breadth, and synthesis.The cited approaches include learnable retrieval decisions, hierarchical and multi-agent coordination, and controlled corpora for retrieval and reasoning.Source: §4 · p. 14

Grounding deep search in multimodal evidence

  • Multimodal deep search connects visual clues with external Web information, with methods and benchmarks addressing image, video, and egocentric-video evidence.In these settings, interpreting a visual observation can change subsequent searches and use of resulting evidence.Source: §4 · p. 14

Acquiring evidence through interactive access

  • Interactive-access research studies website traversal, dynamic interfaces, browser-agent reliability, and coordination across pages or sites.These efforts treat navigation, interface manipulation, and artifact processing as steps that may be needed before source evidence becomes usable.Source: §4 · p. 14

5. Conclusion

OpenDS-World evaluates real-world Web deep search using diverse evidence and exposes persistent execution challenges among evaluated agents.

5. Conclusion

  • OpenDS-World’s 200 tasks span four modalities, 42 formats, and seven access types, and evaluations of 12 frontier models identify lengthy trajectories, costly tool failures, and frequent budget exhaustion for several models.The strongest model achieves 39.00% accuracy, while natural termination does not reliably imply a correct answer.Source: §5 · p. 14

B.1. Benchmark Composition

OpenDS-World spans eight domains, varied anchor-video durations, and multi-source evidence chains. Its composition combines broad domain coverage with tasks that often require inspecting long videos and connecting several verified sources.

B.1. Benchmark Composition

  • 33 tasks each cover Science and Society, the largest domain counts among the 200 tasks; each of the eight domains accounts for 9.0%–16.5%.Daily Life contributes 30 tasks, Culture 27, Nature 22, History 19, and Economy and Technology 18 each.Source: §B.1 · p. 19
  • 56.0% of tasks begin with videos lasting at least ten minutes, including 34.5% with videos of twenty minutes or longer.The remaining duration groups are under two minutes (18.0%), two to five minutes (12.0%), and five to ten minutes (14.0%).Source: §B.1 · p. 19
  • 55.5% of tasks require at least four annotated source entries, and the benchmark contains 787 verified entries, averaging 3.94 per task.44.5% of tasks contain three entries, 28.0% contain four, 20.0% contain five, and 7.5% contain six or more.Source: §B.1 · p. 19

B.2. Evidence Attribute Diversity

OpenDS-World tasks commonly combine multiple evidence modalities, resource formats, and access operations. Its annotations also include a long tail of specialized file types, and its taxonomy describes access requirements independently of the tools used.

B.2. Evidence Attribute Diversity

  • 82.5% of tasks require evidence spanning at least two information modalities, most commonly two modalities (70.5%).17.5% of tasks use one modality, 11.0% use three, and 1.0% use all four; the counts exclude the separately supplied anchor video.Source: §B.2 · p. 19, Figure 11 · p. 20
  • 96.0% of tasks involve at least two resource formats, and 41.0% involve three or more.HTML, PDF, and CSV may all contain text but require different access and parsing mechanisms.Source: §B.2 · pp. 19–20, Figure 11 · p. 20
  • 95.5% of tasks involve at least two access types, and 47.5% involve four or more.Four access types are the most common configuration (30.0%); two tasks require all seven types.Source: §B.2 · p. 20, Figure 11 · p. 20

B.3. Long-Tail File Formats

  • 108 long-tail source entries (13.72% of the 787 annotated entries) span structured data, spreadsheets, scientific datasets, documents, geospatial data, archives, and 3D assets.Their varied structures can require Web navigation, downloading, and specialized parsing.Source: §B.3 · p. 20, Table 2 · p. 21

B.4. Access Requirement Taxonomy

  • The seven access types classify operations for obtaining evidence: observation, hypermedia transitions, parameterized updates, temporal indexing, graphical manipulation, declarative queries, and artifact transfer.Labels can combine for one source, and task-level coverage counts the union of labels across annotated entries; therefore, percentages may exceed 100% when summed.Source: §B.4 · p. 21, Table 3 · p. 21
  • The taxonomy is defined by access operations rather than specific tools; one tool may support multiple types, and one operation may require multiple tool calls.Parameterized updates change explicit inputs or selections, while graphical manipulation changes a spatial or visual representation.Source: §B.4 · p. 21

B.5. Source Requirements Across Difficulty Groups

Evidence composition differs across human-calibrated difficulty groups, though not every source requirement rises monotonically with difficulty. High-difficulty tasks show more audio, ZIP, hypermedia-transition, and artifact-transfer coverage, but less image and video coverage than Low tasks.

B.5. Source Requirements Across Difficulty Groups

  • Audio coverage rises from 7.4% in Low to 22.4% in High, while image coverage falls from 48.1% to 38.8% and video from 40.7% to 30.6%.The difficulty groups are defined by human solving success: Low (54 tasks), Medium (61), and High (85).Source: §B.5 · p. 22
  • ZIP coverage increases from 1.9% in Low to 12.9% in High, while formats grouped as Other rise from 31.5% to 48.2%.Coverage percentages are calculated within each human-calibrated group.Source: §B.5 · p. 22
  • Hypermedia-transition coverage increases from 61.1% in Low to 72.9% in High, and artifact-transfer coverage rises from 53.7% to 67.1%.Temporal indexing stays approximately 47%–48% across all three groups; other access types show non-monotonic patterns.Source: §B.5 · p. 22

C. Detail Experiment Setup

The evaluation uses a unified environment with 16 tools and prompts that leave research strategy to the agents. The harness supports video inspection, manages budgets and context, and separates research termination from final-answer generation.

C.1.1. Tool Setting

  • The environment provides 16 tools, persistent browser interaction, access to downloaded artifacts, and video-frame sampling and subtitle extraction.The prompt leaves search strategy, source selection, and tool-call sequences to the models.Source: §C.1.1 · p. 23, §C.3.1 · p. 24

C.1.2. Input Videos Process

  • Agents receive a question and a read-only local input video, which they can inspect globally or within selected intervals using frame sampling or timestamped subtitles.The default video reader samples 10 frames, supports 1–100 frames per call, and uses midpoints of equally spaced intervals.Source: §C.1.2 · p. 23, §C.3.1 · p. 24

C.2.1. Budget Management

  • The harness manages research budgets and configuration parameters, with visual observations managed separately and older images replaced by omission markers when limits are exceeded.Historical experiments used image limits of 250; the actual setting depends on the run configuration.Source: §C.2.1 · p. 23

C.2.2. Context Management

  • Before each model request, deterministic compaction preserves the four most recent tool observations and truncates older text content to 2,000 characters.Older search-result fields retain only their first two entries; other fields are unaffected by these limits.Source: §C.2.2 · p. 23

C.2.3. Termination Mechanism

  • Research ends when the model gives a complete, nonempty answer without further tool calls, or because of budget exhaustion, incomplete output, context overflow, or execution errors.Termination reason, answer-generation status, and answer correctness are recorded or determined separately.Source: §C.2.3 · pp. 23–24
  • If research ends without a complete answer, a separate finalization stage uses collected evidence without tools and allows at most two requests within a 5,000-token budget.Only nonempty, complete responses without tool calls are accepted; the first request receives 4,000 tokens.Source: §C.2.3 · p. 24

C.3.1. Search Prompts

  • The research prompt directs agents to inspect media, verify evidence, and continue until evidence is sufficient or the budget is exhausted, without prescribing a search strategy.Unless the task specifies another format, agents should give one direct English paragraph of one to three sentences, at most 80 words.Source: §C.3.1 · p. 24

C.3.2. Final Answer Prompt

  • The final-answer prompt requires a short answer based only on collected evidence, forbids further tool calls, and instructs the model to acknowledge insufficient evidence.The default format is one to three English sentences, at most 80 words.Source: §C.3.2 · p. 25

C.4. LLM as Judge Evaluation

The answer judge compares candidate answers with reference answers for semantic equivalence and completeness, using the question only to identify what must be answered. The experiments section analyzes model performance across domains, evidence attributes, and video duration.

C.4. LLM as Judge Evaluation

  • The judge evaluates candidate answers against reference answers for semantic equivalence and completeness, without independently solving questions or using external tools.An absent answer is represented as (empty), and execution_error is null when no error is recorded.Source: §C.4 · p. 25
  • The judge receives the question, reference answer, candidate answer, and execution status, then returns a JSON correctness decision, confidence, and brief reason.Numbers, named entities, dates, directions, and list membership must be correct; refusals, empty answers, insufficient evidence, and failed executions are incorrect.Source: §C.4 · p. 25

D. More Experiments

  • The experiments section analyzes model performance across task domains, evidence attributes, and video duration.Source: D. More Experiments · p. 25

D.1. Domain Performance Analysis

Model accuracy varies across OpenDS-World’s eight task domains. Nature is relatively strong for most models, while model strengths and cross-model variation differ by domain.

D.1. Domain Performance Analysis

  • Accuracy patterns differ by domain: GPT-6 Astra reaches 51.5% in Society and 50.0% in Nature, while Claude Opus 5.5 reaches 50.0% in Nature and 45.5% in Science.Gemini 3.8 Flash reaches 44.4% in both Culture and Economy and leads History at 36.8%.Source: §D.1 · pp. 25–26, Figure 13 · p. 26
  • Nature yields relatively high accuracy for most models, while Science and Society show the largest cross-model standard deviations at 13.2 and 12.9 percentage points.Source: §D.1 · p. 26, Figure 13 · p. 26

D.2. Evidence Attributes Performance Analysis

Model strengths vary across evidence modalities, resource formats, and access requirements. Accuracy also varies across video-duration groups, but those differences do not establish a duration–difficulty relationship.

D.2. Evidence Attributes Performance Analysis

  • GPT-6 Astra leads Text (38.69%) and Image (48.31%), while Gemini 3.8 Flash leads Video (43.66%) and shares top Audio accuracy (50.00%) with Claude Fable 5.1.Different models show relative strengths across the four evidence modalities.Source: §D.2 · p. 26, Table 6 · p. 27
  • GPT-6 Astra leads six of eight format categories, while Gemini 3.8 Flash leads MP4 (42.86%) and MP3 (51.61%); JPG is generally weaker and ZIP varies from 0.00% to 57.14%.The format comparison covers eight categories, including HTML, MP4, PDF, MP3, JPG, CSV, ZIP, and Other.Source: §D.2 · p. 26, Table 7 · p. 27
  • GPT-6 Astra leads six access categories, including Declarative Query (61.54%), while Gemini 3.8 Flash leads Temporal Indexing (46.32%).Declarative Query accuracy ranges from 0.00% to 61.54% across the 12 models.Source: §D.2 · pp. 26–28, Table 8 · p. 28

D.3. Video Duration Analysis

  • Accuracy varies across five anchor-video-duration groups, but the observed differences do not establish that video duration causes task difficulty.The groups contain 36, 24, 28, 43, and 69 tasks, respectively.Source: §D.3 · p. 28

E. Discussion

OpenDS-World tasks can require extensive Web exploration and interaction beyond their annotated evidence chains. The benchmark’s execution traces also expose budget exhaustion and failed-call costs as potential improvement targets.

E. Discussion

  • Tasks with 3, 4, 5, and at least 6 annotated sources have median tool-call counts of 74, 56, 72, and 82.5, respectively, and calls do not rise monotonically with source count.Mean calls range from 42.8 to 89.9 across models, versus 3.94 annotated source entries per task; trajectories also cluster near the 100-call budget.Source: E. Discussion · p. 28, Figure 15 · p. 29
  • Tool-budget exhaustion terminates 15.5%–80.0% of task trajectories across models, while failed calls consume 19.3%–61.9% of cumulative tool-execution latency.OpenDS-World includes 200 answer-verified tasks, 787 verified source entries, 42 file formats, and seven access requirements, providing execution feedback for studying strategy refinement.Source: E. Discussion · p. 30, Figure 16 · p. 30

Related papers

Loading 2610.30181v1…