PaperlayerView exact revision on arXiv

Source-linked AI summary

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen

arXiv:2608.20357v1cs.IRcs.AI

TL;DR

Deep-search systems can fail on underspecified queries, but existing evaluation does not adequately measure whether clarification improves downstream utility without intent leakage. Clarify-Then-Search introduces a closed-book, end-to-end benchmark with static evidence-grounded nuggets, finding consistent gains from clarification while exposing answerability failures and backend-dependent scope.

  • Problem

    Existing clarification benchmarks emphasize question-level properties, while retrieval metrics and leakage-prone end-to-end evaluations incompletely measure downstream deep-search utility.

  • Method

    The benchmark evaluates clarify→rewrite→deep-search pipelines on 518 curated Baidu-derived query pairs using closed-book agents and static weighted, evidence-grounded golden nuggets.

  • Results

    Clarification improves over the no-interaction baseline at k=1 and generally gains with larger budgets; GPT-5.2 leads at k=1, while ERNIE-4.5-Turbo-128K leads overall at k=3.

  • Takeaways & Limitations

    The benchmark supports reproducible, leakage-resistant analysis of clarification utility, question answerability, and budget effects in deep search.

  • Takeaways & Limitations

    The evaluation fixes WebDancer as the deep-search backend, uses an LLM-based unknown policy, and may contain judging noise from borderline semantic matches.

Abstract

from arXiv · show

Deep search is brittle on underspecified user queries: missing constraints such as time, location, scope, or definitions can lead to retrieval drift and incomplete answers. We introduce Clarify-Then-Search, a benchmark for evaluating whether LLM-generated clarification questions improve downstream deep-search utility. Built on real-world query data from the Baidu search engine, the benchmark contains 518 curated instances, each with an intent query and a corresponding underspecified query. For each intent query, we run WebDancer once to archive evidence and construct a static golden reference as weighted, evidence-grounded nuggets with traceable source identifiers. At evaluation time, a Clarifier asks k in {1, 2, 3} questions; a closed-book User Answerer replies only with information explicitly stated in the intent query, otherwise returning unknown; and a closed-book Rewriter produces a rewritten query using only the underspecified query and the elicited question-answer pairs. WebDancer then executes on the rewritten query, and we score end-to-end utility using restore_score_100, a weighted nugget-recall score with partial credit against the static gold. Across all evaluated models, clarification improves over the no-interaction baseline at k=1, and larger budgets generally yield further gains. GPT-5.2 achieves the highest mean score at k=1, while ERNIE-4.5-Turbo-128K becomes the overall top-performing model at k=3. Diagnostics reveal a consistent failure mode: many systems over-ask region-only questions that are often unanswerable from the intent and thus elicit unknown. Clarify-Then-Search enables leakage-resistant and reproducible evaluation of clarify-then-search pipelines, with fine-grained analyses of question utility, answerability, and budget effects in deep search.

1 Introduction

Clarify-Then-Search addresses the difficulty of evaluating whether clarification improves deep-search utility under underspecified queries and realistic information constraints. It introduces a leakage-resistant benchmark with reproducible nugget-based evaluation, diagnostic signals, and consistent gains from clarification.

  • Underspecified deep-search queries can omit temporal, geographic, entity, numeric, or definitional constraints, causing planning errors, wasted tool calls, off-target evidence, and incomplete or incorrect answers.
  • Prior benchmarks often measure question fluency or relevance rather than downstream search utility, while retrieval metrics miss effects on planning, browsing, aggregation, and answer completeness.
  • The benchmark evaluates clarification through an end-to-end clarify→rewrite→deep-search pipeline using a closed-book protocol that prevents rewriting stages from observing oracle intent.
  • 518 curated Baidu-derived instances pair underspecified queries with underlying intent queries and provide standardized evaluation outputs and reproducibility artifacts.
  • All Clarifiers improve over the no-interaction baseline at k=1, and larger clarification budgets generally yield further gains; GPT-5.2 leads at k=1, while ERNIE-4.5-Turbo-128K leads overall at k=3.
  • Diagnostics identify region-question bias as a failure mode because such questions often elicit unknown under the closed-book constraint.

2 Related Work

The benchmark builds on research in clarification, query rewriting, tool-augmented agents, and nugget-based evaluation. It integrates these foundations into a leakage-resistant, end-to-end deep-search benchmark scored against static evidence-grounded gold.

  • Clarifying Questions in Conversational Search: Prior clarification resources study how systems ask questions for underspecified conversational-search queries and support retrieval-oriented or multi-turn evaluation.
  • Query Rewriting for Conversational Search: Conversational-search rewriting research converts contextual or ambiguous inputs into explicit standalone queries, supported by datasets such as CANARD and QReCC.
  • Tool-Augmented Agents and Deep Search Evaluation: Tool-augmented-agent research motivates evaluation of reasoning, web search, browsing, API use, and realistic interactive environments.
  • Nugget-Based and Evidence-Grounded Evaluation: Nugget-based evaluation scores coverage of atomic information units, with weighting and evidence matching supporting fine-grained answer-completeness assessment.
  • Synthesis: Clarify-Then-Search integrates these foundations into a closed-book benchmark on 518 underspecified queries, measuring downstream nugget restoration against static evidence-grounded gold.

3 Task Definition and Closed-Book Protocol

The task evaluates whether clarification can transform an underspecified query into a retrieval-ready query without leaking the hidden intent. A closed-book interaction separates intent-grounded answering from rewriting, while fixed evidence-grounded nuggets provide the evaluation target.

  • Task Definition: WebDancer executes the rewritten query and produces an answer with evidence traces, which are evaluated against a static golden reference constructed for the intent query.
  • Baselines and Targets: The Orig baseline runs WebDancer directly on the underspecified query, while Gold runs it once on the intent query to create the fixed evaluation target.
  • Closed-Book Protocol: The User Answerer can access the hidden intent only to answer the current question, while the Rewriter sees only the underspecified query and elicited question-answer pairs.
  • Closed-Book Protocol: When the intent does not specify an attribute, the User Answerer returns unknown; the Rewriter preserves that ambiguity rather than inventing a constraint.
  • Leakage Prevention: Any added rewritten-query constraint must be grounded in observed question-answer pairs rather than copied from the hidden intent.
  • Static Golden Reference: Static golden nuggets are atomic, judgeable statements tied to archived evidence snippets by source identifiers, enabling reproducible comparisons across Clarifiers and budgets.

4 Dataset

The dataset pairs real-world underspecified queries with hidden intent queries and evaluates whether clarification restores weighted, evidence-grounded answer nuggets. Static gold references and fixed scoring support reproducible comparisons across systems and clarification budgets.

  • Dataset construction: The benchmark contains 518 information-seeking instances pairing each underspecified blurred_query with an underlying intent fused_query.Pairs are constructed from real user intents by removing constraints such as time, region, entity scope, and criteria.
  • Evaluation target: Clarification questions are evaluated by whether they elicit answerable information that improves downstream deep-search utility, rather than against canonical question labels.This treats clarification questions as system outputs and avoids prescribing a single correct clarification.
  • Static golden nuggets: For each intent query, one WebDancer run archives the final answer and search/visit traces as the evidence pool for static gold construction.The archived traces support reproducible golden-reference creation.
  • Static golden nuggets: An evidence-grounded extractor produces atomic, judgeable nuggets with supporting evidence identifiers, required sub-questions, missing-evidence requirements, unsupported claims, and integer weights w_j∈{1, 2, 3}.The resulting nugget set is reused across evaluated systems and clarification budgets.
  • Scoring: restore_score_100 compares each candidate answer with static golden nuggets using weighted nugget recall with partial credit.Coverage is judged as full, partial, or none, and mean restore_score_100 is reported alongside p50, p90, minimum, and maximum.

5 Experimental Setup

The experimental setup isolates clarification quality in a controlled closed-book pipeline: only the Clarifier varies, while answering, rewriting, search, gold construction, and judging remain fixed. Systems are tested on 518 selected underspecified queries across budgets k∈{1, 2, 3}.

  • Controlled components: The pipeline varies only the Clarifier while fixing the User Answerer, Rewriter, deep-search backend, nugget construction, and nugget judging.This design attributes performance differences primarily to the ambiguity dimensions asked about and the usefulness of elicited answers.
  • Clarifier: Clarifiers generate k∈{1, 2, 3} questions from the blurred query alone, targeting dimensions such as time, region, entity scope, or definition.Questions are produced with deterministic decoding and strict JSON-array formatting.
  • Closed-book interaction: The fixed User Answerer returns an answer only when the intent explicitly specifies the requested attribute; otherwise it returns unknown without fabrication.The Rewriter receives only the blurred query and Q&A pairs, cannot add entities or constraints, and preserves ambiguity for unknown answers.
  • Deep-search execution: WebDancer runs on each rewritten query with fixed configuration and tool budget, producing a candidate answer and tool traces for evaluation.The candidate is scored against static golden nuggets using restore_score_100, with distributional statistics also reported.
  • Evaluation scope: The benchmark evaluates 518 released instances selected to represent underspecified queries where clarification is expected to matter most, without main-paper difficulty subsets.WebDancer is fixed deliberately to compare Clarifiers within one deep-search environment rather than claim backend-independent absolute scores.

6 Results

Under closed-book evaluation on 518 instances, one-turn clarification improves restore_score_100 over orig, while larger budgets generally produce further gains. Diagnostics show that unknown answers and region-only questions limit the utility of clarification, although multi-turn interactions reduce no-signal cases.

  • Main Results: One-Turn Clarification: All Clarifiers improve restore_score_100 over orig at k=1, with GPT achieving the highest mean and median gains.Open-weight models gain +3.09 to +4.03, while GPT, Claude, and Gemini gain +6.44 to +7.04.
  • Main Results: One-Turn Clarification: Paired bootstrap 95% confidence intervals for improvement over orig at k=1 are strictly above zero.This indicates that the observed improvements are not driven by a small number of outliers.
  • UA Diagnostics: Unknown Rate and Question-Type Bias: At k=1, ua_unknown_rate remains 0.600–0.701, so roughly two thirds of questions yield no intent-grounded constraint for rewriting.When the answer is unknown, the Rewriter has little grounded content to incorporate and the rewritten query remains close to the underspecified input.
  • UA Diagnostics: Unknown Rate and Question-Type Bias: Region-only questions have ua_unknown_rate 0.750–0.869, while cq_region_rate ranges from 0.143 for Kimi to 0.498 for Gemini.The results distinguish plausible, topical questions from clarification questions that yield usable information under the closed-book protocol.
  • Budgeted Clarification: k=2 and k=3: Increasing the budget from k=1 to k=2 yields mean gains of +6.16 to +7.64 over orig for all models.At k=3, ERNIE achieves the largest improvement (+8.90), although some models saturate or regress slightly relative to k=2.
  • Budgeted Clarification: k=2 and k=3: GPT's all_unknown rate falls from 0.618 at k=1 to 0.375 at k=2 and 0.189 at k=3.Larger budgets increase the chance of eliciting at least one useful constraint, aligning with restoration gains.

7 Analysis

The analysis shows that clarification utility depends on eliciting answerable, grounded information rather than merely plausible questions. Larger budgets usually help, but gains and model rankings depend on question selection, while robustness checks preserve the main improvement pattern.

  • Information Gain is the Main Bottleneck: At k=1, UA unknown rates remain 0.600–0.701 across models, leaving Rewriters with little grounded content to resolve residual ambiguity.This information bottleneck helps explain both reliable improvement over orig and remaining headroom.
  • Information Gain is the Main Bottleneck: GPT’s all_unknown rate falls from 0.618 at k=1 to 0.375 at k=2 and 0.189 at k=3.Additional turns mainly help by increasing the chance of eliciting at least one usable constraint.
  • Question-Type Bias is a Measurable Failure Mode: Region-only questions have UA unknown rates of 0.750–0.869 despite cq_region_rate ranging from 0.143 for Kimi to 0.498 for Gemini.Their low yield makes plausible-sounding regional clarification a measurable failure mode.
  • Question-Type Bias is a Measurable Failure Mode: Across larger budgets, region-only unknown rates remain high, so repeatedly asking for location does not reliably increase information gain without location information in the intent.The benchmark therefore exposes low-yield question types beyond surface fluency or topical relevance.
  • Budget Effects: Mean improvements increase from k=1 to k=2 for all models, while k=3 further improves several systems but is not monotonically beneficial for every model.Extra turns help when they elicit additional grounded constraints rather than repeat unanswerable ambiguity dimensions.
  • Budget Effects: GPT performs best at k=1 and k=2, whereas ERNIE becomes strongest at k=3, showing that different budgets reward different clarification abilities.Evaluating only one budget can therefore misrepresent sustained question-selection quality.
  • Ask-or-Not Decisions: High all_unknown and question-type-specific unknown rates provide measurable signals for deciding whether to ask or switch to retrieval-first exploration.The benchmark supports gating policies based on predicted answerability and expected information gain.
  • Robustness: At k=1, GPT > ERNIE > orig remains stable across ERNIE-, GPT-, and Claude-based judges despite changes in absolute score scale.This indicates that the main end-to-end conclusion is not an artifact of one coverage judge.

8 Conclusion

Clarify-Then-Search evaluates clarification under realistic information constraints using closed-book interaction and static evidence-grounded nuggets. Clarification consistently improves over no interaction, larger budgets generally strengthen restoration, and ERNIE-4.5-Turbo-128K leads overall at k=3.

  • Conclusion: Clarification consistently improves over the no-interaction baseline, with larger clarification budgets generally yielding stronger restoration.The benchmark measures restoration after clarification, rewriting, and deep search.
  • Conclusion: ERNIE-4.5-Turbo-128K becomes the overall top-performing model at k=3.The conclusion also identifies unknown or low-yield answers, especially from region-only questions, as persistent diagnostic issues.

Limitations

The benchmark’s conclusions are bounded by its closed-book answerer, fixed WebDancer backend, and LLM-based nugget construction and judging. These choices support controlled evaluation but leave policy, backend generality, and borderline-match noise as limitations.

  • Protocol and Evaluation Scope: The LLM-based User Answerer’s exact unknown policy can affect measured utility, even though the closed-book design prevents intent leakage to the Rewriter.The protocol therefore constrains both information flow and evaluation outcomes.
  • Protocol and Evaluation Scope: Fixing WebDancer makes the benchmark a controlled evaluation under a shared deep-search pipeline rather than a backend-agnostic ranking of all search systems.The scope boundary concerns generalization across deep-search backends.
  • Protocol and Evaluation Scope: LLM-based golden-nugget construction and coverage judging with deterministic decoding may still introduce noise for borderline semantic matches.Strict rubrics reduce but do not eliminate this judging limitation.

B Evaluation Prompt Summary

The evaluation prompts operationalize the benchmark’s evidence-grounded scoring procedure. One prompt generates traceable golden nuggets from retrieved sources, and another labels candidate coverage as full, partial, or none.

  • Golden Reference Generation: Prompt D generates evidence-grounded static golden nuggets from retrieved sources for traceable scoring.The benchmark releases the full executable prompts and JSON schemas.
  • Nugget Coverage Judging: Prompt E judges each golden nugget against a candidate answer using full, partial, or none coverage labels.The evaluation prompt requires strict semantic consistency and critical-detail coverage.

C Additional Diagnostics

The appendix adds diagnostics omitted from the main text, expanding region-only question statistics and summarizing supporting-backbone stability for the UA and Rewriter components.

  • The appendix provides additional diagnostics that were omitted from the main text because of space limits.
  • Table 10 reports turn-level UA unknown rates conditioned on region-only questions across 518 benchmark instances.
  • Table 11 summarizes backbone stability using UA agreement, rewrite similarity, and unknown preservation.UA agreement compares known/unknown status across supporting backbones; rewrite similarity measures normalized query-text similarity; unknown preservation tracks avoidance of concretizing unknown answers.
Loading 2608.20357v1…