PaperlayerView exact revision on arXiv

Source-linked AI summary

When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu

arXiv:2606.27669v2cs.CL

TL;DR

Real-world search queries are often ambiguous, but existing benchmarks largely assume they are complete and explicit. DiscoBench evaluates clarification-aware deep search and finds that proactive clarification outperforms repeated searching or direct guessing, while ambiguity detection and effective clarification remain distinct challenges.

  • Problem

    Existing deep-search benchmarks assume complete, explicit queries despite vague or fragmented requests that can create cascading errors across multi-step reasoning chains.

  • Method

    DiscoBench models ambiguity across multi-step checkpoints and evaluates multi-turn search agents using a progressive-clue user simulator across task utility, detection, strategy, and cost.

  • Results

    Proactive clarification is substantially more effective than repeated search or direct guessing, while ambiguity detection and effective clarification remain distinct capabilities.

  • Takeaways & Limitations

    Future search agents need stronger ambiguity awareness and interaction planning alongside retrieval and reasoning abilities.

  • Takeaways & Limitations

    DiscoBench covers four objective ambiguity types and relies on an LLM-based simulator that may not capture the diversity and unpredictability of real human interactions.

Abstract

from arXiv · show

Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.

1 Introduction

Deep-search agents face cascading errors when vague initial queries remain unresolved, yet existing benchmarks largely omit proactive clarification. DISCOBENCH addresses this gap by evaluating ambiguity detection, clarification, and interaction costs during multi-step search.

  • Motivation: Deep search amplifies subtle query ambiguities through multi-step reasoning chains, causing cascading navigation and information-integration errors.Failing to clarify ambiguous checkpoints can waste computational resources by sending the search down the wrong path.
  • Benchmark gap: Existing retrieval benchmarks assume explicit queries and emphasize multi-hop reasoning, while ambiguity-focused datasets do not evaluate proactive clarification during search.This leaves search-agent disambiguation capabilities insufficiently assessed.
  • DISCOBENCH: DISCOBENCH contains 211 samples and 463 ambiguity instances across 11 real-world domains and four ambiguity types.It models ambiguity as a dynamic phenomenon propagating along multi-step reasoning trajectories rather than as a static query property.
  • Evaluation framework: The benchmark evaluates ambiguity detection, clarification effectiveness, and interaction cost through an ambiguity-aware multi-turn framework with a user simulator.The simulator progressively reveals discriminative clues during interaction.
  • Findings: Experiments show that ambiguity detection and effective clarification are distinct capabilities, and repeatedly searching instead of asking can yield lower success rates than direct guessing.Even stronger models often fail to recognize ambiguity or ask effective clarification questions.

2 Related Work

Prior search-agent benchmarks emphasize reasoning depth, information width, or realistic web navigation, with newer work beginning to combine depth and width. Related ambiguity and interactive benchmarks address uncertainty and collaboration separately, but static evaluations often prioritize answer identification over dynamic agent–user clarification.

  • Search-agent benchmarks: Search-agent benchmarks traditionally target either multi-hop reasoning depth or broad information synthesis, while DeepWideSearch begins to encompass both dimensions.GAIA and BrowseComp represent depth-focused benchmarks; PaSa, SPAR, and WideSearch represent width-focused benchmarks.
  • Search-agent benchmarks: WebArena, VisualWebArena, Mind2Web, and WebShop evaluate web navigation capabilities in realistic environments.
  • Ambiguity benchmarks: AmbiEnt, AmbigQA, and ASQA develop taxonomies for semantic, structural, and multi-answer factual ambiguity, but rely on static evaluation centered on answer identification.These benchmarks do not directly evaluate the dynamic, interactive process of resolving uncertainty through agent–user collaboration.
  • Interactive benchmarks: Interactive benchmarks span code generation, travel planning, implicit intent understanding, conflicting requests, and conversational clarification of coreference or semantic vagueness.Examples include ColBench, UserBench, IN3, GAIA2, and Abg-CoQA.

3 Task Formulation

The task formulates multi-turn interactive retrieval as sequential question answering over ordered checkpoints, where agents detect ambiguity and choose between continued retrieval and user clarification. Checkpoints are unambiguous or ambiguous, and performance is assessed through utility, detection, interaction strategy, and cost efficiency.

  • Question and Checkpoints: Each question q is decomposed into an ordered sequence of n checkpoints, each representing an intermediate retrieval sub-goal.Checkpoints are denoted {CP1, CP2, . . . , CPn}.
  • Question and Checkpoints: Checkpoints are classified as Unambi, which permits direct retrieval, or Ambi, which produces multiple candidates or no valid result.Ambi checkpoints contain one of four injected ambiguity types.
  • Agent Actions and User Interaction: At each checkpoint, the agent chooses among three actions, including searching, answering, and asking the user for clarification.For an ambiguous checkpoint, ASK(·) releases a pre-defined clue c, after which the agent refines its search and issues ANSWER(r∗).
  • Evaluation: The framework evaluates agents on four aspects: task utility, ambiguity detection, interaction strategy, and cost efficiency.These dimensions cover both retrieval outcomes and interactive behavior.

4 Methodology of Dataset Construction

DISCOBENCH is constructed through two phases: preparing verified multi-hop seed questions, then injecting resolvable ambiguity and adding discriminative facts for interactive clarification. The benchmark spans 11 domains and four ambiguity types, with quality controlled through manual review.

  • Two-Phase Construction: The pipeline has two phases: Seed Data Preparation builds multi-hop reasoning chains, while Ambiguous Data Construction injects ambiguity and generates discriminative facts.This transforms deterministic question answering into interactive reasoning tasks requiring clarification.
  • Seed Data Preparation: Seed topics are manually collected across 11 knowledge domains using encyclopedic resources and search-engine results, with the benchmark primarily constructed in Chinese.Sources include Wikipedia, Baidu Baike, Google, Bing, and Baidu.
  • Seed Data Preparation: LLMs preliminarily expand seed topics into factual QA pairs and graph-structured candidate chains, which humans verify and reconstruct into high-quality multi-hop questions.The framework combines automated generation and external retrieval with human verification.
  • Ambiguous Data Construction: Ambiguity points are retained when entities have similar alternatives, downstream reasoning remains executable under underspecification, and one user clue can resolve the ambiguity.Candidate positions are manually verified before ambiguity is injected.
  • Ambiguous Data Construction: Candidate entities are rewritten using shared attributes, while retrieval-augmented LLMs generate discriminative facts across attributes, time, relations, numerical facts, versions, and organizations.Facts are manually verified for factual correctness, and all samples undergo checks for retrieval feasibility, logical consistency, and ambiguity solvability.
  • Dataset Statistics: The dataset contains 211 samples and 463 ambiguity instances spanning Entity, Version, Criteria, and Factual Inaccuracy ambiguity types.Difficulty is defined by ambiguity checkpoints: easy has 1, medium 2, and hard 3.

5 Experiments

Experiments show that frontier models remain weak at clarification-aware deep search, while guided prompting improves ambiguity detection more than downstream reasoning. Success depends on effective clarification and external retrieval rather than simply searching more, with ambiguity itself remaining a major difficulty source.

  • Experimental setup: The evaluation measures task utility, ambiguity detection, interaction quality, and cost efficiency under neutral and guided prompting settings.The evaluated models use the same interactive retrieval framework and checkpoint-level evaluator.
  • Overall performance: 43.1% is Doubao-Seed-2.0-Pro’s best neutral end-to-end accuracy, while most models remain below 40%.Gemini-3.1-Pro reaches 40.8%, whereas MiniMax-M2.7 and Qwen3.6-Max achieve 16.1% and 12.3%, respectively.
  • Overall performance: Guided prompting raises average end-to-end accuracy from 28.6% to 33.7% and detection F1 from 45.3% to 64.9%.Checkpoint pass rate also rises from 50.1% to 57.6%; the gains mainly reflect better ambiguity detection, and stronger local interaction can still fail to recover the full reasoning trajectory.
  • Interaction behavior: Detection and clarification effectiveness are distinct: Qwen3.6-Max has 16.0% detection F1 but achieves 94.7% CE-A and 89.5% CE-B.It asks only 0.07 questions per task under neutral prompting, indicating strong conditional question quality but weak proactive detection.
  • Interaction behavior: 93.4% is the average pass rate for SearchThenAsk, exceeding DirectGuess at 56.5% and SearchHeavyGuess at 51.9%.SearchHeavyGuess underperforms DirectGuess despite more retrieval, because models fail to escalate retrieval uncertainty into clarification.
  • Ablation analysis: Removing external search drops Doubao-Seed-2.0-Pro’s accuracy from 43.1% to 2.4%, while removing ambiguity improves accuracy by 26.8% to 40.2% across models.The ablation indicates that agents need retrieval to gather evidence, verify constraints, and revise trajectories, while ambiguity remains a major difficulty source.

6 Conclusion

DISCOBENCH evaluates clarification-aware deep search by modeling ambiguity as a dynamic issue in multi-step search and testing agents’ ability to clarify and recover reasoning paths. Experiments show that interactive capabilities remain a major limitation despite improved ambiguity detection through guided prompting.

  • 6 Conclusion: DISCOBENCH is a benchmark for evaluating clarification-aware deep search.It models ambiguity as a dynamic issue emerging during multi-step search.
  • 6 Conclusion: Structured checkpoints test whether agents detect ambiguity, ask for clarification, and recover correct reasoning paths using user-provided clues.
  • 6 Conclusion: Guided prompting improves ambiguity detection, but current LLM-based search agents still struggle with interactive deep search and limited end-to-end performance.
  • 6 Conclusion: Proactive clarification is substantially more effective than repeated search or direct guessing.
  • 6 Conclusion: Future search agents need stronger retrieval and reasoning alongside better ambiguity awareness and interaction planning.

Limitations

DiscoBench is limited to four ambiguity types in objective question answering, leaving subjective preference ambiguity underexplored, and its multi-turn user simulator relies on LLM-generated rather than real human behavior.

  • DiscoBench covers four representative ambiguity types grounded in objective question answering, while subjective preference ambiguity remains underexplored.The paper leaves more complex ambiguity forms for future work.
  • The benchmark’s progressive-clue user simulator generates interaction behavior with LLMs rather than real human users.This limits how directly the simulator represents human interaction.

Ethical Considerations … B.4 Clarification Effectiveness

The paper documents DISCOBENCH’s research-only construction and contributions, then defines question- and checkpoint-level evaluation metrics for accuracy, ambiguity detection, and clarification effectiveness. Clarification metrics distinguish asking the right question from using returned clues to advance.

  • Ethical Considerations: DISCOBENCH uses public web resources without private or personally identifiable information and is intended solely to evaluate ambiguity handling and clarification in search agents.Its LLM-based user simulator may not fully reflect the diversity of real-world interactions.
  • A Author Contributions: The authors jointly designed the benchmark and methodology, led data construction and annotation, implemented the evaluation framework, conducted experiments, analyzed results, wrote the paper, and supervised the project.The listed responsibilities span benchmark design, multi-hop and ambiguity data, quality control, implementation, experimentation, writing, figures, revision, and supervision.
  • B Evaluation Metrics: All evaluation metrics are computed at the question or checkpoint level and averaged over valid questions for each model.This establishes the common aggregation framework for the reported metrics.
  • B.1 End-to-End Accuracy: End-to-end accuracy tests whether the final answer matches ground truth using an LLM-based equivalence judge that abstracts away surface-form variation.Question-level normalization gives each valid question equal weight in the model-level score.
  • B.2 Checkpoint Pass Rate: Checkpoint pass rate counts a checkpoint as advanced when the agent answers it correctly and proceeds, or correctly completes the final checkpoint.The model-level rate averages question-level scores, so questions contribute equally regardless of checkpoint count.
  • B.3 Ambiguity Detection Metrics: Ambiguity detection compares checkpoint type with interaction behavior using TP, FN, FP, and TN across ambiguous and non-ambiguous checkpoints.Detection accuracy measures overall decision correctness, while F1 balances precision and recall when checkpoint classes are unevenly distributed.
  • B.4 Clarification Effectiveness: Clarification effectiveness uses checkpoints where the agent invokes ASK as the denominator for both CE-A and CE-B.CE-A measures whether the question targets the right ambiguity; CE-B measures whether a correct clarification helps advance the checkpoint.

C Additional Analysis by Reasoning Effort

Increasing reasoning effort consistently improves overall performance, especially ambiguity-related metrics, but high effort alone does not overcome substantial limitations in clarification-aware deep search.

  • Ambiguity-related metrics: 10.1 points: Ambi. Rec. improves from 37.2% to 47.3%, exceeding the gain on CP.Det. F1 also increases by 9.0 points, showing particularly strong improvements on ambiguity-related metrics.
  • Mechanisms: Higher reasoning effort mainly helps models identify ambiguous search states, compare candidate entities, and incorporate user clues into later search refinement.These capabilities are central to navigating multiple ambiguity checkpoints while maintaining and revising the search trajectory.
  • Limitations: Even with high reasoning effort, accuracy remains below 45% and Ambi. Rec. remains below 50%, indicating that increased effort alone is insufficient.DiscoBench requires both final-answer retrieval and continuous evidence sufficiency judgments across multi-step ambiguity checkpoints.

D Additional Analysis by Ambiguity Complexity

Under neutral prompting, ambiguity complexity substantially degrades search-agent accuracy from Easy to Hard across nearly all models. The decline reflects challenges in retrieval, ambiguity recognition, clarification initiation, and avoiding plausible but incorrect multi-step search trajectories.

  • Performance by ambiguity complexity: Across nearly all evaluated models, accuracy consistently decreases from Easy to Hard, showing that ambiguity complexity adds substantial difficulty beyond ordinary retrieval and reasoning.Figure 7 reports performance across ambiguity-complexity levels under neutral prompting.
  • Performance by ambiguity complexity: Even stronger models achieve relatively strong Easy performance but still show marked accuracy drops on Hard examples.The affected models include Doubao-Seed-2.0-Pro (High), Gemini-3.1-Pro, Claude-Opus-4.7, and DeepSeek-V4-Pro.
  • Failure modes: As ambiguity becomes subtler and accumulates across checkpoints, models increasingly follow plausible but incorrect search trajectories without initiating clarification.The performance gap indicates that stronger reasoning ability alone is insufficient, while lower-performing models show the same downward trend from a lower baseline.

E Profile Classification Details · F Evaluated Models and API Configurations · G Token Consumption

The paper classifies ambiguous-checkpoint trajectories by retrieval and clarification timing, documents the evaluated models’ reproducibility configurations, and reports token consumption across main and ablation settings.

  • E Profile Classification Details: Four behavioral profiles distinguish DirectGuess, SearchHeavyGuess, DirectAsk, and SearchThenAsk trajectories by search count and clarification timing.The threshold is K =3, defined as the median search count among successful no-ask trajectories.
  • E Profile Classification Details: DirectAsk is extremely rare, appearing in only 0–7 trajectories per model, with 7 of 9 models recording N ≤1.This indicates that current models almost never initiate clarification before retrieval.
  • E Profile Classification Details: Common-subset model rankings remain highly consistent with the full dataset, with Spearman ρ=0.95 across 146 ambi-CPs.The subset includes ambi-CPs reached by all 9 models; GPT-5.4 and Qwen are excluded for the stated reach-related reasons.
  • F Evaluated Models and API Configurations: The API configuration records each model’s paper name, provider, invocation identifier, and reasoning- or thinking-mode setting for reproducibility.Configurable-effort models use the highest available setting in the main experiments, while the table defines shorthand such as xhigh, high, medium, thinking, and adapt.
  • F Evaluated Models and API Configurations: All SEARCH calls use Tavily, while Gemini-3-Flash-Medium at medium thinking level serves as both simulated user and checkpoint-level judge.The auxiliary model releases discriminative clues during clarification turns and scores each step.
  • G Token Consumption: Token consumption is reported for evaluated models under Neutral and Guided prompting, with GPT-5.4’s Guided entry omitted because those runs lacked sufficient valid results.Doubao-Seed-2.0-Pro-Medium appears only in the Neutral reasoning-effort analysis.
  • G Token Consumption: Additional token-consumption results cover search-tool removal and unambiguous-question ablations under neutral prompting.These runs are reported separately from the main Neutral/Guided comparison.

H Reproducibility under a Black-Box Search Backend

DISCOBENCH uses Tavily as a black-box search backend, so retrieval results can vary across days and runs even when the underlying questions remain solvable. This variability affects agents’ search trajectories and efficiency, requiring reported scores and errors to be interpreted as backend-dependent outcomes.

  • Backend behavior: Tavily’s undisclosed index, ranking model, and freshness policies make SEARCH results vary across days in snippets, rankings, and source domains.Variation also reflects the web’s non-stationarity, including pages appearing, changing, or being de-indexed.
  • Solvability: DISCOBENCH questions remain solvable because their gold answers are stable, verifiable facts supported by widely indexed public sources.The backend changes the retrieval surface rather than whether the underlying answer is answerable.
  • Variance across runs: Two runs of the same agent on one question may follow different trajectories and use different numbers of SEARCH calls before relevant evidence appears, despite both succeeding.The black-box effect changes which snippets are surfaced and ranked, affecting which queries suffice and retrieval efficiency.
  • Interpretation and replication: DISCOBENCH scores should be interpreted as expectations over Tavily snapshots rather than deterministic per-run quantities.Individual errors should be inspected to distinguish retrieval ordering from genuine ambiguity-handling failures before attributing them to model capability.

I Examples … L Prompt Templates

The paper illustrates DISCOBENCH’s ambiguity cases, documents annotation and quality-control procedures, and specifies probing, user-simulation, and guided-agent prompts for clarification-aware evaluation.

  • I Examples: Three cases demonstrate cascading Entity ambiguity, combined Factual Inaccuracy and Version ambiguity, and Criteria ambiguity across multi-hop trajectories.The cases span Video Games, Sports, and Technology domains and require agents to adapt clarification behavior within trajectories.
  • J Annotation Details: Annotators and inspectors were undergraduate students recruited across institutions and compensated piece-rate, with $39,000 paid for the annotation effort.They consented to public release, and the task used public web resources without collecting personal or sensitive information.
  • K Quality Inspection: 314 candidate samples entered multi-stage quality control, with structural validation, LLM probing, ambiguity assessment, manual review, and rule-based final decisions.Closed-book probing tests knowledge leakage, while open-book probing permits at most 25 search calls but prohibits clarification questions.
  • K Quality Inspection: 236 (75.2%) of 314 candidate samples passed quality control, while 78 (24.8%) were removed; the final common benchmark contains 211 samples.Samples were removed when they did not effectively require both deep retrieval and multi-turn clarification.
  • QC PROBING PROMPTS: QC probing contrasts closed-book answers from training data with open-book answers using search while forbidding clarification and capping WebSearch/WebFetch calls at 25.The prompts require direct JSON outputs and instruct agents to guess rather than decline in closed-book testing.
  • L.1 Multi-Turn Responder Prompt MULTI_TURN_RESPONDER_PROMPT: The multi-turn responder prompt enforces progressive disclosure, exposing only the current checkpoint and releasing forbidden information indirectly through clue_if_asked.Its decision matrix distinguishes ambiguity_detected, wrong_ambiguity_detected, correct_answer, missed_ambiguity, search_only, and task completion outcomes.
  • L.3 Guided System Prompt GUIDED_SYSTEM_PROMPT: The guided system prompt instructs agents to search first, ask when multiple candidates remain or retrieval cannot resolve the issue, and answer only after unique identification.Clarification questions should target one missing discriminative dimension, while every turn must end with either ask or answer.
Loading 2606.27669v2…