Source-linked AI summary
Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents
Nicholas Edwards, Sebastian Schuster
TL;DR
Coding agents often face underspecified software-engineering tasks but are optimized for autonomous execution. This paper evaluates an uncertainty-aware multi-agent scaffold that detects missing context separately from code execution, achieving a 69.40% resolve rate and calibrated clarification-seeking behavior.
Problem
Existing coding agents are optimized for autonomous completion rather than interactive collaboration when software-engineering instructions contain missing context.
Method
The paper evaluates uncertainty-aware single- and multi-agent scaffolds on an underspecified variant of SWE-bench Verified, separating underspecification detection from code execution.
Results
69.40% resolve rate was achieved by UA-MULTI across Claude Sonnet 4.5 and Kimi K2.6, closing the gap with fully specified-task performance.
Takeaways & Limitations
The findings provide evidence that frontier LLM agents can identify missing information and seek clarification selectively according to task complexity.
Takeaways & Limitations
The evaluation uses an LLM-based user simulator that may not reflect the nuance and variability of real human users.
Abstract
from arXiv · showhide
As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that decouples underspecification detection from code execution. Across both proprietary and open-weight frontier LLMs, our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated information-seeking behavior, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.
1 Introduction
The paper studies whether LLM agents can recognize underspecified software-engineering tasks and seek clarification to maintain alignment with user goals. It evaluates single- and multi-agent frameworks on an underspecified SWE-bench Verified variant and finds interactive agents can achieve resolve rates comparable to autonomous agents given fully specified issues.
- Motivation: Unlike static or single-turn information-gathering settings, software-engineering agents must explore large repositories while remaining aligned with implicit developer intentions.Existing agents are primarily optimized for autonomous completion rather than interactive collaboration.
- Evaluation: The study evaluates clarification seeking in a dynamic, multi-turn software-engineering framework built from an underspecified SWE-bench Verified variant.Information is removed from the original GitHub issues, and agents decide when to query users using their own calibration of external uncertainty.
- Contributions: Interactive agents can identify and retrieve missing information, achieving a resolve rate comparable to autonomous agents given fully specified issues.The contribution evaluates frontier LLMs including Claude Sonnet 4.5 and Kimi K2.6.
2 Method
The study evaluates clarification-seeking agents on an underspecified SWE-bench Verified variant using OpenHands, two frontier LLM backbones, and a simulated user. It compares autonomous, interactive, single-agent uncertainty-aware, and multi-agent uncertainty-aware configurations.
- Evaluation Dataset: The evaluation uses CMU LTI (2024), an underspecified SWE-bench Verified variant containing 500 human-annotated GitHub issues with important details removed while repository terminology is preserved.The dataset construction uses GPT-4o to summarize fully specified issues into underspecified variants.
- Agent Framework: All experiments use OpenHands, which lets agents iteratively navigate repositories, edit files, execute scripts, and debug code in a secure sandbox, with a 100-iteration limit.The iteration limit ensures comparability with Vijayvargiya et al. (2026b).
- Agent Backbones: Experiments use Claude Sonnet 4.5, a proprietary model, and Kimi K2.6, an open-weight model, as frontier coding-agent backbones.Both models are selected as competitive proxies for current interactive software-engineering capabilities.
- Evaluation Configurations: The study compares FULL, HIDDEN, and INTERACTIVE BASELINE configurations, differing in issue completeness, user access, and whether querying is forced by the prompt.GPT-5.1 serves as the simulated user and receives the original fully specified issue.
- Uncertainty-Aware Scaffolds: The proposed uncertainty-aware scaffolds detect missing context without a hardcoded interaction prompt: UA-SINGLE uses one coding agent, while UA-MULTI separates execution from detection with Main and Intent Agents.The Intent Agent produces a binary clarification decision, and detection constrains the Main Agent’s next action to query the user; both agents share a backbone.
3 Results and Discussion
UA-MULTI substantially improves resolve rates over single-agent uncertainty handling for both Claude Sonnet 4.5 and Kimi K2.6, while calibrating clarification behavior to task difficulty. Its interaction strategy is selective and more effective than simply asking questions broadly, though prompt-based interaction can be brittle.
- Overall Resolve Rates: 69.40% resolve rate: UA-MULTI significantly outperforms UA-SINGLE for Claude Sonnet 4.5 and closes the gap with INTERACTIVE BASELINE and FULL.For Claude Sonnet 4.5, UA-SINGLE reaches 61.20% versus 54.80% for HIDDEN and 70.40% for INTERACTIVE BASELINE; UA-MULTI’s comparisons are supported by p < 0.001, p = 0.621, and p = 0.458, respectively.
- Overall Resolve Rates: 69.40% resolve rate: UA-MULTI significantly improves over UA-SINGLE for Kimi K2.6, approaches FULL, while INTERACTIVE BASELINE falls to 47.20% because of instruction-following failures.The Kimi comparison reports UA-SINGLE at 61.60%, with UA-MULTI’s improvement at p < 0.001 and a remaining significant gap from FULL at p < 0.05; failures involved misusing the task completion tool to ask questions.
- Agent Uncertainty is Calibrated to Task Difficulty: UA-MULTI asks selectively and resolves queried Claude Sonnet 4.5 tasks at 65.99%, versus 55.81% for UA-SINGLE and 44.48% for HIDDEN.It queried fewer tasks than UA-SINGLE (Nask=344 vs. Nask=369), while INTERACTIVE BASELINE achieved comparable queried-task resolution but asked in nearly every instance.
- Agent Uncertainty is Calibrated to Task Difficulty: UA-MULTI better differentiates clarification needs by task complexity, with Claude Sonnet 4.5’s medium-task ask rate 9.28% higher than its easy-task rate.UA-SINGLE’s corresponding increase is 2.43%, while Kimi K2.6 shows similar easy-to-medium query-rate calibration between agents.
- Proactive Information Seeking: UA-MULTI increases proactive querying, averaging 3.06 queries per Claude task and 8.71 for Kimi K2.6, with timing concentrated in early-to-middle or middle execution stages.Claude UA-SINGLE averages 1.84 queries per task; UA-MULTI distributes Claude queries across early and middle execution, while Kimi queries mainly occur in the middle stage.
4 Conclusion
The study presents an uncertainty-aware multi-agent scaffold that enables LLM agents to identify missing information and seek clarification in underspecified software-engineering tasks. It achieves a 69.40% resolve rate and exhibits calibrated information-seeking, while clarification behavior varies with underlying model capability.
- Conclusion: 69.40% resolve rate was achieved across proprietary and open-weight LLM backbones, effectively closing the gap with an autonomous agent operating on fully specified tasks.The result was obtained on an underspecified variant of SWE-bench Verified.
- Conclusion: The uncertainty-aware multi-agent scaffold isolates underspecification detection so agents can independently identify missing information and seek clarification.This design targets collaboration in underspecified software-engineering tasks.
- Conclusion: The scaffold shows well-calibrated information-seeking by recognizing resolvable issues and avoiding unnecessary interaction on simpler tasks.Both LLM backbones benefit from the scaffold, but Kimi K2.6 has higher query rates and poorer calibration, indicating that clarification behavior depends on underlying model capability.
Limitations
The study’s limitations concern the reliability of LLM-based user simulation, dependence on tailored multi-agent prompting, and the cost and generalization constraints of frontier-model evaluation.
- User Simulator: LLM-based user simulators may be unreliable proxies for human behavior because they can be unnaturally cooperative and lack real-user nuance and variance (Naous et al., 2026; Seshadri et al., 2026).Although strict guardrails were implemented and responses were generally reasonable, results may vary with human users.
- Prompting and Training: The strongest approach depends on tailored prompts for multiple agents, although successful interaction trajectories could potentially train single models through finetuning or reinforcement learning.The results also suggest frontier models possess latent capacity for uncertainty monitoring and proactive clarification without specialized prompting.
- Model Generalization and Cost: The scaffold generalized across Claude Sonnet 4.5 and Kimi K2.6, but frontier-model evaluation incurred non-trivial financial costs and smaller open-weight models may lack calibration for interactive underspecification (Vijayvargiya etThe cost is detailed in Appendix F.
Ethical Considerations · A Related Work
The paper situates its clarification-seeking scaffold within proactive dialogue, ambiguous code-generation, and uncertainty-detection research while noting limits on generalization and the environmental and financial costs of multi-agent deployment.
- Ethical Considerations: The scaffold’s results should not be extrapolated beyond SWE-bench Verified to high-stakes settings such as security-critical applications without further experimentation.SWE-bench Verified covers only a subset of software engineering tasks, and reliable detection of missing information elsewhere remains unestablished.
- Ethical Considerations: Multi-agent deployment increases inference overhead, creating environmental and financial costs that must be weighed against developer-productivity gains.The paper identifies this trade-off in its cost breakdown in Appendix F.
- Ethical Considerations: The authors used an AI assistant for experimental code generation and writing revision, while retaining responsibility for the research and verifying all AI outputs.The authors state that original writing, conceptualization, methods, experiments, and analyses were performed by them.
- A Related Work: Clarification-seeking research in NLP addresses proactive dialogue for shared understanding, open-domain question answering, conversational search, and preference elicitation (Clark and Wilkes-Gibbs, 1986; Rao and Daumé III, 2018).The cited literature spans dialogue-based tasks including open-domain question answering, conversational search, and preference elicitation.
- A Related Work: Prior methods teach clarification through conversation-trajectory utility modeling (Zhang et al., 2025), simulated self-play (Andukuri et al., 2024), and uncertainty guidance (Testoni and Fernández, 2024).These approaches target when questions are necessary and how models can improve clarification skills.
- A Related Work: Coding benchmarks often evaluate isolated-function edits with only one predetermined clarification round or short, constrained exchanges (Li et al., 2023).The paper characterizes robust handling of ambiguous or underspecified instructions as a major challenge for software-engineering agents.
- A Related Work: Recent coding-agent training uses reinforcement learning (Sun et al., 2025) and contrastive training, but some frameworks rigidly restrict questions to the trajectory’s beginning (Vijayvargiya et al., 2026a; Darji and Lutellier, 2025).These frameworks defer code execution until the initial prompt is disambiguated, despite missing information sometimes emerging during codebase exploration.
- A Related Work: Unlike rigid initial-questioning frameworks, the paper’s multi-agent scaffold decouples underspecification detection from code execution and enables questions throughout a trajectory.This design addresses missing information that may become apparent only after initial exploration of a codebase.
B Dataset Construction and Validation
The evaluation uses an automatically generated underspecified SWE-bench Verified variant, validated through corpus comparison and manual spot-checking. The spot-check results and baseline behavior indicate that removed information meaningfully determines when interaction is needed.
- Dataset Construction: The dataset directly adopts Vijayvargiya et al. (2026b)’s underspecified SWE-bench Verified variant, automatically generated by prompting GPT-4o to summarize original GitHub issues while retaining repository terminology and important technical details.The evaluation uses this dataset as described in Section 2.1.
- Dataset Validation: Vijayvargiya et al. (2026b) applied distributional difference analysis (Zhong et al., 2023) to compare generated issues with naturally occurring underspecified issues, which typically retain more concrete technical details.Examples include code snippets, specific error messages, and file/line references.
- Empirical Spot-Check Analysis: In a random sample of 10 instances, essential information was removed in 5 cases, while the other 5 retained enough technical detail to likely localize the files requiring edits.The essential information included OS configurations and user-provided value constraints; retained details included specific class or method names.
- Empirical Spot-Check Analysis: 54.80% versus 70.80%: Claude Sonnet 4.5 solved none of the 5 instances missing essential information but solved all 5 retaining sufficient technical details, matching the HIDDEN–FULL performance gap.This pattern indicates that the dataset provides signal for identifying tasks requiring interaction to resolve underspecification.
C Task and Scaffold Prompts · D Question Analyses
The task prompts distinguish fully specified, underspecified, and explicitly interactive settings, while the uncertainty-aware scaffolds use prompts to detect missing information and trigger clarification. The user simulator adds environment-specific guidance and limits interactions to test efficient information-seeking.
- C.1 Task Prompts: The FULL baseline receives the original fully specified issue, whereas HIDDEN and both UNCERTAINTY-AWARE scaffolds receive an underspecified issue.
- C.1 Task Prompts: Only the INTERACTIVE BASELINE receives an augmented prompt explicitly stating that information is missing and requiring questions before execution.
- C.2 Agent Scaffold Prompts: UNCERTAINTY-AWARE (SINGLE) repeatedly prompts the agent to assess underspecification and use the clarify tool when missing information is detected.
- C.2 Agent Scaffold Prompts: UNCERTAINTY-AWARE (MULTI) assigns an Intent Agent to inspect state history, identify missing information, and decide when the Main Agent must pause for clarification.
- C.2 Agent Scaffold Prompts: An augmented Intent Agent prompt was tested to reduce Kimi K2.6’s over-querying by suppressing underspecification flags when users cannot provide more details or request continued work.
- C.3 User Simulator Prompt: The user simulator prompt was augmented with environment-specific guidance to prevent misleading advice or unintended information leakage in the OpenHands SWE-bench evaluation.
- C.3 User Simulator Prompt: Simulator rules clarify that agents should not modify SWE-bench test files and that edits in /workspace are tested separately from /testbed.
- C.3 User Simulator Prompt: The simulated user allows three interaction turns per trajectory, then instructs the agent to proceed; only the INTERACTIVE BASELINE is warned of this limit upfront.
D.1 Question Statistics
Question statistics show that uncertainty-aware agents ask fewer but more iterative and targeted questions than the interactive baseline, with query behavior varying substantially by backbone and timing. Their questioning also occurs at different stages of trajectories, reflecting differences in when underspecification is detected and, for Kimi K2.6, instruction-following failures.
- Interaction amount: The interactive baseline queried nearly all Claude Sonnet 4.5 tasks (Nask=496, 1.02 Q/Task) but far fewer Kimi K2.6 tasks (Nask=194, 1.00 Q/Task).For Kimi K2.6, instruction-following failures caused the model to use the task completion tool for questions instead of initiating proper query turns.
- Interaction amount: UA-MULTI initiated queries in 344 Claude Sonnet 4.5 tasks and 435 Kimi K2.6 tasks, averaging 3.06 and 8.71 questions per task, respectively.The higher Kimi K2.6 query rate reflects a tendency to over-flag underspecification.
- Interaction length: For Claude Sonnet 4.5, UA-MULTI produced the shortest queries and answers at 171.57 and 173.65 tokens, whereas the interactive baseline produced 251.33- and 415.53-token exchanges.The contrast supports a targeted multi-agent questioning strategy versus asking for potentially missing information upfront.
- Temporal distribution: For Claude Sonnet 4.5, the interactive baseline asked 97.6% of questions early, while UA-SINGLE skewed late at 43.9%.The late skew suggests UA-SINGLE often attempts code modifications before recognizing that information is missing.
- Temporal distribution: For Kimi K2.6, the interactive baseline concentrated 66.5% of questions in the middle of trajectories, contrasting with Claude Sonnet 4.5’s early concentration.This pattern likely reflects instruction-following failures that disrupted early-stage querying.
D.2 Query Frequency by Task Difficulty … E Underspecified Issue Examples
The agents generally increased querying as task difficulty rose, with stronger calibration for Claude Sonnet 4.5 than Kimi K2.6. Conditional outcomes, interaction transcripts, and underspecified issue examples further show how clarification behavior and information removal shape evaluation.
- D.2 Query Frequency by Task Difficulty: The difficulty analysis maps querying behavior to SWE-bench Verified’s human-annotated levels, which categorize issues by estimated human developer time to fix.Table 3 presents ask rates for UA-SINGLE and UA-MULTI by difficulty.
- D.2 Query Frequency by Task Difficulty: UA-MULTI showed stronger difficulty sensitivity than UA-SINGLE, with a 9.28% versus 2.43% ask-rate increase between tasks estimated at under 15 minutes and those taking 15 minutes to 1 hour.Both scaffolds reached a 100% interaction rate on the hardest tasks, estimated to require more than 4 hours.
- D.2 Query Frequency by Task Difficulty: For Kimi K2.6, UA-MULTI queried 82.99%–100% of tasks across difficulty levels, while UA-SINGLE’s ask rate followed a non-monotonic pattern.The consistently high UA-MULTI query rate suggests reduced sensitivity to task complexity, whereas UA-SINGLE declined from the medium-difficulty level described in the passage.
- D.3 Conditional Resolve Rates for UNCERTAINTY-AWARE (SINGLE): Table 4 reports resolve rates conditioned on whether UA-SINGLE queried the user at least once, using Nask and Nnot to denote the corresponding subset sizes.The table identifies the best per-subset results in bold.
- D.3 Conditional Resolve Rates for UNCERTAINTY-AWARE (SINGLE): For Claude Sonnet 4.5, UA-SINGLE resolved 216 of 369 queried tasks, compared with 175 for HIDDEN on the same subset.On the 131 tasks where UA-SINGLE did not query, it resolved 90 tasks versus 99 for HIDDEN, indicating that querying was most useful when missing information was critical.
- D.3 Conditional Resolve Rates for UNCERTAINTY-AWARE (SINGLE): For Kimi K2.6, UA-SINGLE’s resolve rates were nearly unchanged between queried tasks (60.48%) and non-queried tasks (62.16%), indicating weaker calibration.The passage contrasts this pattern with Claude Sonnet 4.5 and notes that UA-SINGLE’s clarification behavior was less calibrated than UA-MULTI in other configurations.
- D.4 Qualitative Example of Agent Interaction: In the pytest-dev__pytest-7324 example, uncertainty-aware agents asked more specific technical questions than the baseline, and UA-MULTI uniquely issued a mid-trajectory follow-up query.The task concerned resolving a pytest-related Python interpreter crash, and the passage identifies the transcript comparison as Figure 6.
- E Underspecified Issue Examples: Figure 7 compares original and underspecified descriptions for two tasks, where summarization generally removed code snippets, file paths, line references, and stack traces while retaining high-level issue terminology.Appendix B’s spot-check analysis notes that the extent of information reduction varied across instances.
F Computing Costs · User Simulator Prompt
The multi-agent scaffold more than doubles inference cost yet remains financially negligible per task, while the user simulator and interaction examples operationalize guarded, iterative clarification for underspecified coding tasks.
- F Computing Costs: More than doubling inference cost still yields a negligible absolute financial cost per task for the multi-agent scaffold.The authors characterize this expenditure as a favorable trade-off because resolving underspecification can save human debugging time and effort.
- F Computing Costs: The uncertainty-aware scaffolds use distinct reminder and specialized Intent Agent prompts to guide single- and multi-agent behavior.Figure 4 presents the reminder prompt used at each turn for the single-agent scaffold and the system prompt for the multi-agent Intent Agent.
- User Simulator Prompt: The user simulator prompt includes guardrails against unintended test modifications and environment-directory mismatches.These safeguards are specified as rules 5 and 6 in Figure 5.
- User Simulator Prompt: On pytest-dev__pytest-7324, both interactive baseline and uncertainty-aware single-agent setups identify missing information through one upfront query turn.The uncertainty-aware single-agent queries are more specific and technical than the interactive baseline’s generic queries.
- User Simulator Prompt: On the same task, the multi-agent framework adds late-stage iterative clarification after a test failure, unlike the single-agent setups.This demonstrates interaction that continues beyond the initial clarification exchange.
- User Simulator Prompt: Missing OS information makes pylint-dev__pylint-7080 difficult without clarification, whereas preserved class and method names let scikit-learn__scikit-learn-26323 be resolved autonomously.The comparison contrasts underspecification that blocks completion with identifiers sufficient for issue localization and resolution.