Source-linked AI summary
How Specific Do I Need to Be? Effects of User Query Underspecification on Deep Research Agents
Yi Wen, Xiangjue Dong, Bingxuan Li, Hangxiao Zhu, Leo Zhicheng Liu, Yu Zhang, James Caverlee
TL;DR
Deep research agents are typically evaluated on fully specified queries, although users may not be able to express all their information needs and open-ended reports are difficult to verify. UNDERSPEC-DR evaluates agents using controlled, realistic query omissions and analyzes both answers and search behavior. Across the evaluated agents, underspecification reduces fulfillment of user needs while agents’ search behavior relies less on the original query and more on learned priors.
Problem
Because open-ended research reports are difficult to verify, the effects of underspecified queries on deep-research answer quality and ambiguity resolution remain unclear.
Source: §1 · p. 1Method
UNDERSPEC-DR creates controlled query variants based on omissions found in researchers’ conversations and evaluates agents’ answers and intermediate search behavior.
Source: §1 · p. 1, §3.1 · p. 3Results
Takeaways & Limitations
The findings support developing agents that recognize unresolved information gaps and seek user input before proceeding with research.
Source: §8 · pp. 8–9The evaluation covers three agents with accessible search and tool-use trajectories, excluding closed-source systems that do not expose them.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Deep research agents (DRAs) conduct long-horizon reasoning, search, and literature synthesis to provide long-form answers to complex, domain-specific questions. These agents are typically trained and evaluated on fully specified user queries. However, real-world user queries can often be underspecified. We introduce UNDERSPEC-DR, a framework for systematically evaluating how underspecified queries affect agents' final answer quality and intermediate search behavior. Across three DRAs we evaluated, underspecification substantially reduces how well DRA's answers fulfill users' information needs, even though they remain grounded in cited evidence. Our analysis of intermediate search behavior suggests a possible explanation: DRAs conduct broader searches that draw less on the original user query but rely more on their learned prior during training, which can be misaligned with users' intent. Our findings reveal the vulnerability of current DRAs to underspecified user queries and inform future efforts to build DRAs that proactively recognize when additional user input is needed to reliably complete deep research tasks.
1 Introduction
Deep research agents are built and evaluated on fully specified queries, although users may struggle to articulate their information needs and open-ended reports are difficult to verify. UNDERSPEC-DR evaluates answer quality and search behavior under realistic query omissions, finding that underspecification can reduce user-need fulfillment without increasing hallucination.
1 Introduction
- UNDERSPEC-DR evaluates answer quality and intermediate search behavior using realistic omissions of scope, terminology, or user preferences.The framework draws these omission types from researchers’ initial–final query pairs and creates matched variants of existing deep-research benchmarks.Source: §1 · pp. 1–2
- The study compares three agents on complete and underspecified queries, assessing rubric satisfaction, citation support, recognition of missing information, and search behavior.The motivating gap is that long, open-ended reports are difficult to verify, making the effects of underspecification and ambiguity resolution hard to measure.Source: §1 · pp. 1–2
- Underspecification reduces how well answers satisfy query-specific rubrics, even when only one key detail is omitted, but does not increase hallucinations.Removing scope constraints or obscuring technical terminology produces substantial mean score reductions across most agents and benchmarks; preference omissions generally have smaller, often nonsignificant effects.Source: §1 · p. 2
- Agents often rely more on learned priors than on eliciting missing information, and may proceed without asking users for clarification.Their searches also rely less on the original query and more on parametric knowledge.Source: §1 · p. 2
- Underspecified queries prompt broader, more diffuse searches and more previously unseen documents, without greater use of retrieved content in later searches.The authors propose that agents should recognize unresolved information gaps and make them explicit before proceeding.Source: §1 · pp. 2–3
2 Related Work
Prior work examines underspecification and ambiguity resolution in LLMs and agentic settings, often where task success is readily verifiable. For open-ended deep research, answer quality and clarification behavior remain harder to assess, especially across long-horizon tool use.
2 Related Work
- Prior studies examine omitted requirements and ambiguity-resolution methods in standalone LLMs and agentic tasks, including software engineering and question answering.Many such settings offer verifiable outcomes, such as requirement satisfaction or passing code tests.Source: §2 · p. 3
- Open-ended research reports are difficult to verify, leaving the effects of underspecification on deep-research answers unclear.This contrasts with tasks where failures or deviations from expected outcomes provide clearer signals that clarification may be needed.Source: §2 · p. 3
- Long-horizon agents must decide whether uncertainty can be resolved through further environmental interaction or requires additional information from the user.Prior work explores clarification policies and uncertainty-informed decisions, but extending these approaches to long-horizon agentic settings remains underexplored.Source: §2 · p. 3
3 The UNDERSPEC-DR Framework
UNDERSPEC-DR formalizes query underspecification and measures its effects on answer quality and search behavior. It derives omission categories from real researcher conversations, creates matched benchmark variants, and checks their naturalness and quality through model and human review.
3.1 Problem Formulation
- Answer-quality change compares scores for answers to an underspecified variant and the complete query, evaluating both against the original information need.The framework also defines changes in intermediate search behavior through measures of search trajectories.Source: §3.1 · p. 3
3.2 Deriving Dimensions of Underspecification
- Analysis of 176 researcher query pairs identifies three common omissions: scope, terminology, and preferences, which together cover 94% of the pairs.The categories cover missing scope (45%), terminology (37%), and preferences (33%); 21% of pairs lack more than one type.Source: §3.2 · p. 4
3.3 Constructing Matched Variants
- The authors use rubric-guided LLM rewriting to create matched underspecified variants for three deep-research benchmarks.A discrimination test found GPT-4o identified generated variants with 58% and 54% accuracy across two runs, only slightly above chance, suggesting similarity to natural underspecified queries.Source: §3.3 · p. 4
3.4 Human Validation of Synthetic Query Variants
- Human reviewers checked whether edits removed the target information, preserved other content, and left queries natural and coherent.They revised 6% of variants for naturalness and 8% to ensure technical terminology was appropriately obscured.Source: §3.4 · pp. 4–5
4 Experimental Setup
The study evaluates three deep research agents on open-ended research benchmarks, measuring both answer quality and search behavior. It also assesses how well answers use and support claims with retrieved evidence.
Benchmarks and answer quality metrics
- Three open-ended research benchmarks were used: ResearchQA, ScholarQA-Bench subsets, and DeepResearch Bench.Gemini 3.1 Pro scored rubric-based coverage using each benchmark’s evaluation protocol.Source: §4 · p. 5
- Citation-quality metrics were limited to DR Tulu-8B and GPT-5.4 with tools on ScholarQA-Bench and DeepResearch Bench because Tongyi-DR gives no explicit citations.Document utilization measures cited documents relative to retrieved documents; citation precision measures cited documents that support their claims.Source: §4 · pp. 5–6
Search behavior metrics
- Search behavior was characterized by query drift, transition types, token provenance, document novelty, and context-driven term adoption, using paired tests and Cohen’s dz.Transition types include Specialization, Generalization, Exploration, and Repetition; token provenance tracks whether terms come from the user query, earlier searches, retrieved documents, or none of these.Source: §4 · p. 6
5 RQ1: How Does Underspecification Affect Answer Quality?
Underspecification reduces rubric-based answer coverage, especially when scope constraints are removed or technical terminology is obscured. The results show no significant decline in citation precision or document utilization.
RQ1: How Does Underspecification Affect Answer Quality?
- Rubric-based coverage declines for many underspecified queries, with significant mean reductions for scope removal and terminology obscuring across most agents and benchmarks.Removing user preferences generally produces smaller, often nonsignificant reductions; scores decline further as queries become more underspecified.Source: DRAs cannot reliably infer users’… · p. 6
- Underspecification produces no statistically significant change in citation precision or decline in document utilization, which increases in three of eight settings.Together with reduced rubric coverage, the findings indicate lower alignment with users’ information needs without evidence of increased hallucination among cited claims.Source: DRAs cannot reliably infer users’… · p. 6
- Agents with higher scores on complete queries generally incur larger absolute score losses under full underspecification.DR Tulu-8B and Tongyi-DR outperform GPT-5.4 with tools on complete queries but appear more dependent on fully specified user needs.Source: DRAs cannot reliably infer users’… · p. 7
Underspecification in User Queries?
DRAs often fail to recognize missing information and rarely ask users to clarify it. When they do recognize ambiguity, they usually proceed under assumptions, which do not guarantee better answers.
Underspecification in User Queries?
- Unaware responses, in which agents proceed without explicitly recognizing underspecification, are the most frequent category across DRAs.The analysis distinguishes Unaware, Assume (recognize underspecification and proceed with assumptions), and Ask (request clarification).Source: Underspecification in User Queries? · p. 7
- When agents acknowledge ambiguity, they usually proceed under stated assumptions rather than ask for clarification.Tongyi-DR shows the least explicit awareness, while DR Tulu-8B and GPT-5.4 with tools more often state assumptions.Source: Underspecification in User Queries? · p. 7
- Assume runs have larger answer-quality losses than Unaware runs in three of four conditions, so stating assumptions does not guarantee better answers.The authors suggest that recognized ambiguity may reflect greater uncertainty about user intent, making clarification more appropriate than proceeding under assumptions.Source: Underspecification in User Queries? · p. 7
7 RQ3: How Does Underspecification Affect Intermediate Search Behavior?
Underspecification shifts search behavior toward broader, less user-grounded exploration. Agents retrieve more new documents without correspondingly increasing their use of retrieved content in subsequent searches.
RQ3: How Does Underspecification Affect Intermediate Search Behavior?
- Under the most underspecified variant, Exploration transitions increase and Specialization transitions decrease across all three DRAs.Obscuring technical terminology also pronouncedly affects search strategy; changes in Generalization and Repetition are generally smaller and usually nonsignificant.Source: §7 · p. 7, Figure 5 · p. 8
- Document novelty increases under full underspecification for all three DRAs, but context-driven term adoption does not rise and drops significantly for DR Tulu and Tongyi.Novelty also increases without terminology or scope for DR Tulu and Tongyi.Source: Figure 7 · p. 8, §7 · p. 8
- Tokens traceable to the user query decrease significantly while untraced tokens increase significantly in most cases, suggesting greater reliance on parametric knowledge.Scope removal consistently increases untraced-token share more than terminology obscuring; removing preferences does not significantly change token provenance.Source: Figure 6 · p. 8, §7 · p. 8
8 Discussion and Conclusion
The paper introduces UNDERSPEC-DR to study how query underspecification affects deep research agents’ answers and search behavior. It finds that answers can be evidence-grounded yet miss users’ needs, and discusses prompting and agent-training implications.
8 Discussion and Conclusion
- Omitting scope constraints or precise terminology can degrade answer quality and alter search behavior; users should state these details clearly.User preferences generally have a smaller effect on answer quality.Source: §8 · p. 8
- The authors argue that agents should learn to identify missing information and distinguish it from uncertainty that further search can resolve.They note that imitation-learning data capturing both research trajectories and appropriate clarification actions can be difficult to obtain.Source: §8 · p. 8
- UNDERSPEC-DR evaluates how controlled query underspecification affects deep research agents’ answer quality and intermediate search behavior.The framework also examines agents’ awareness of underspecification and their responses to it.Source: §8 · pp. 8–9
Limitations
The evaluation is limited by its language coverage and by which agents expose trajectories for analysis.
Limitations
- Experiments primarily use English queries, with Chinese tasks included through DeepResearch Bench; underspecification may differ across other languages.The authors identify extending the framework to other languages as future work.Source: Limitations · p. 9
- The study covers three agents with accessible search and tool-use trajectories, excluding closed-source systems that do not expose them.Those excluded systems may use different search and clarification strategies.Source: Limitations · p. 9
Ethics Statement
The paper frames underspecified queries as a risk for users and describes procedures for constructing variants, classifying agent responses and search transitions, configuring agents, and collecting real-world queries.
Ethics Statement
- When queries are underspecified, current agents may return well-cited answers to a broader or different question than intended, usually without signaling missing information.The authors note possible risks when users rely on such answers in medical or policy research.Source: Ethics Statement · p. 9
A.1 Constructing Underspecified Query Variants
- The rewriting procedure checks for preference information, terminology, and scope constraints before generating applicable underspecified variants.The prompt examples identify format, depth, audience, and criteria as preferences; proper nouns as terminology; and scenario, domain, time, or location as scope.Source: §A.1 · p. 11
A.2 Underspecification Awareness and Response Classification
- Agent responses are classified as Ask, Assume, or Unaware, distinguishing clarification requests, stated assumptions followed by an answer, and no recognition of ambiguity.The classification prompt considers the original query and the agent’s full response.Source: §A.2 · pp. 11–12
A.3 Search-Query Transition Classification
- Consecutive search-query transitions are classified as Specialization, Generalization, Exploration, or Repetition based on how the current query relates to the previous one.Exploration shifts to another aspect within the same topic; Repetition preserves equivalent intent.Source: §A.3 · p. 12
B Deep Research Agent Configuration
- The three evaluated agents were DR Tulu, GPT-5.4 with tools, and Tongyi-DR, run with a budget of at most 15 tool calls per query.GPT-5.4 used the same framework and tool set as DR Tulu; Tongyi-DR used a different backbone and tool configuration.Source: B Deep Research Agent Configuration · p. 12
C Real-World Query Collection Protocol
- Three junior researchers in electrical engineering, computer science, and construction science shared DR Tulu query histories over two weeks after revising unsatisfactory answers.Because the agent did not support multi-turn interaction, each reformulation was submitted as a new self-contained query.Source: C Real-World Query Collection Protocol · pp. 12–13
D Answer Analysis Protocol
The selected examples show that removing scope or obscuring terminology can reduce rubric coverage, though losses are not inevitable. The section also describes analysis protocols and reports measures of answer quality, clarification, search drift, document use, and citation precision.
D Answer Analysis Protocol
- The answer-analysis protocol sampled about 10% of 1,922 responses, refined labels to Ask, Assume, and Unaware, and validated model annotations against human labels.Human and model annotations had Cohen’s κ of 0.92 before the model was used to annotate all responses.Source: D Answer Analysis Protocol · p. 13
E.1 Lower Rubric Score After Removing Scope from the User Query; E.2 No Decrease in Rubric Score After Removing Scope from the User Query
- 0.8333 to 0.4167: removing the post-tonsillectomy scope shifted an answer toward other bleeding contexts and reduced rubric coverage.In a contrasting case, removing the severity qualifier left coverage at 1.0000 because the answer retained the intended context.Source: §E.1 · p. 13, §E.2 · p. 13
E.3 Lower Rubric Score After Obscuring Terminology in the User Query; E.4 No Decrease in Rubric Score After Obscuring Terminology in the User Query
- For the fungal query, an answer broadened beyond the specified species and rubric coverage fell from 0.7500 to 0.2500; in a zeolite pair, coverage increased from 0.59375 to 0.84375.The zeolite answer still discussed sodium hydroxide, zeolites, and clinoptilolite; this single example does not establish that generalization systematically improves answer quality.Source: §E.3 · p. 14, §E.4 · p. 14
F.1 Answer Quality
- Table 3 reports average rubric-based answer quality across query variants, with significance markers comparing variants against the fully specified condition.Its last column gives relative change from the fully specified query to the fully underspecified query.Source: §F.1 · p. 14, Table 3 · p. 15
F.2 Clarification Behavior
- Table 4 reports the shares of Ask, Assume, and Unaware responses for each agent, benchmark, and query variant.Source: §F.2 · p. 14, Table 4 · p. 15
F.3 Query Drift
- Search-query drift increased significantly across all three agents only under the most underspecified variant, qU; scope or terminology changes significantly increased drift only for DR Tulu-8B.The metric captures semantic displacement between the first and final search queries, not distance from the original user need.Source: §F.3 · p. 14, Figure 8 · p. 15
F.4 Document Utilization
- Table 5 reports mean document utilization (% cited/retrieved) for each underspecified variant versus its paired complete query q, pooled across three benchmarks for DR Tulu-8B and GPT-5.4 with tools.Each comparison includes only questions with that variant answered under both conditions, so the complete-query mean q differs slightly across variants.Source: §F.4 · p. 14, Table 5 · p. 16
F.5 Citation Precision
- Citation precision showed no statistically significant change, including under full underspecification.Table 6 reports paired mean precision as supporting documents divided by cited documents, pooled across ScholarQA-Bench and DeepResearch Bench for answers with citations under both conditions.Source: §F.5 · p. 14, Table 6 · p. 16