LHAW: Controllable Underspecification for Long-Horizon Tasks

• Long-horizon agents must distinguish solvable tasks from those requiring clarification, because each human interaction interrupts execution and adds latency and supervisor switching costs.Silent assumptions may lead to cascading workflow consequences, whereas excessive questions can indicate poor judgment about which gaps matter.Source: §1 · p. 1
• Existing long-horizon benchmarks assess execution with sufficient specifications, while clarification benchmarks use short contexts where asking carries little cost.VitaBench studies complex user interactions but does not systematically evaluate detection of underspecification in long-horizon tasks.Source: §1 · p. 1
• LHAW generates underspecified variants by removing information across four dimensions at configurable severity levels, then uses agent trials to classify their outcomes.The classifications are outcome-critical, divergent, or benign; the pipeline also scores segments for criticality and guessability.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The paper releases 285 variants across three benchmarks and reports that GPT-5.2 over-clarifies while Gemini models under-clarify.It also introduces Gain/Q, performance gain per question, to measure clarification efficiency.Source: §1 · p. 2

2.1 Long-Horizon Workflows

• A long-horizon workflow requires dozens to hundreds of sequential actions across stages, with human interaction available but costly.Clarification costs include interruption, latency while the workflow is blocked, and erosion of trust from excessive questions; integrating these costs with information value is left for future work.Source: §2.1 · pp. 2–3

2.2 Two Types of Ambiguity

• Semantic ambiguity permits multiple valid interpretations, whereas underspecification means critical information is missing and must be acquired for success.LHAW focuses on underspecification and classifies variants as outcome-critical, divergent, or benign based on terminal-state divergence and agent behavior.Source: §2.2 · p. 3

3.1 Task Selection and Evaluation Design

• Tasks are prefiltered for baseline competence, using pass@3 or checkpoint progress when full completion is sparse, so drops track ambiguity sensitivity.THEAGENTCOMPANY tasks use partial completion when full end-to-end success is too rare for practical filtering.Source: §3.1 · p. 3

3.2 Underspecification Taxonomy

• The taxonomy’s four removable information dimensions—Goal, Constraint, Input, and Context—were derived from recurring failure patterns across 33 THEAGENTCOMPANY tasks.Missing expected outputs could yield technically correct but incomplete work, while missing context could prompt systematically incorrect assumptions.Source: §3.2 · p. 4

3.3 Synthetic Underspecification Pipeline

• Segment extraction identifies atomic prompt spans and scores criticality and guessability; candidate generation deletes, vaguifies, or genericizes segments, with more removals increasing severity.The segment priority score is criticality × (1 − guessability); removing more segments can also create a valid new task, motivating empirical validation.Source: §3.3 · p. 5

3.6 Pipeline Validation

• Both 1- and 2-segment removal successfully underspecify tasks; ask_user recovers more performance with two removals, while vague references can prompt confident errors.The authors report that vague references such as “the file” or “an appropriate color” can lead agents to make incorrect assumptions.Source: Table 2 · p. 5, §3.6 · pp. 6–7

3.3 Synthetic Underspecification Pipeline

• Multiple agent trials classify variants as outcome-critical, divergent, or benign from their terminal states, while a New Task filter removes convergent failures on valid substituted tasks.The New Task filter targets cases where all trials fail but converge to the same terminal state.Source: §3.3 · pp. 5–6

3.4 Benchmark Construction

• The benchmark contains 285 final variants: 85 from THEAGENTCOMPANY and 100 each from SWE-BENCH PRO and MCP-ATLAS.THEAGENTCOMPANY’s 85 samples comprise 40 outcome-critical, 30 divergent, and 15 benign variants.Source: §3.4 · p. 6

3.5 User Simulator

• A simulated user exposed through an MCP-hosted ask_user tool answers agent questions about removed segments using the original task, ground truth, and a response policy.Questions outside the removed segments are deferred to the agent’s best judgment.Source: §3.5 · p. 6

3.6 Pipeline Validation

• Across the validation trials, the authors examine whether underspecification produces reliable failure and whether prompt manipulations create predictable difficulty curves.They also compare removal strategies across Gemini-3-Flash; the supplied passage reports substantial outcome-critical ambiguity across strategies.Source: §3.6 · pp. 6–7

4.1 Experimental Setup

• The evaluation uses five frontier models, three independent trials per variant, benchmark-native pass@3 and checkpoint accuracy, and Ask%, Avg Q, and Gain/Q clarification measures.The agent runner replaces well-specified prompts with generated variants and gives ask_user visibility into the original prompts.Source: §4.1 · pp. 7–8

4.2 Value of Information: Agent Behavior across Benchmarks

• Across three benchmarks, underspecification consistently degrades performance, while ask_user enables meaningful but generally incomplete recovery; degradation concentrates in outcome-critical variants.On MCP-ATLAS, Opus-4.5 reaches 78% pass@3 with ask_user, up from 47% without it, but below its 100% baseline.Source: §4.2 · p. 8
• On SWE-BENCH PRO, Sonnet-4.5 leads underspecified pass@3 at 79.0% without ask_user gain, while GPT-5.2 gains 22.0 points to reach 91%.On MCP-ATLAS, Claude Opus-4.5 performs best before and after underspecification; on THEAGENTCOMPANY, GPT-5.2 performs best in the underspecified regime, including with ask_user.Source: §4.2 · p. 8
• GPT-5.2 asks most often and gains most in absolute terms, but Gemini-3-Pro achieves THEAGENTCOMPANY’s highest Gain/Q (0.24) at 18% Ask%, while Sonnet-4.5 leads MCP-ATLAS (0.17).GPT-5.2 records the lowest Gain/Q on MCP-ATLAS and THEAGENTCOMPANY.Source: §4.2 · p. 8

4.3 Cost of Information

• As expected user time cost rises, Gain%/Q increases while checkpoint accuracy falls: the Supervisor setting yields 64.9% Ckpt% and 0.09% Gain%/Q, versus the Executive setting’s 56% and 0.13%.Clarification calls also decline from 0.99 per trial for outcome-critical tasks with the Supervisor to 0.22 for benign tasks with the Executive.Source: §4.3 · pp. 8–9, Table 4 · p. 9

4.4 Failure Modes of Clarifying Questions

• Question Quality, especially compound questions, is the most common failure mode; GPT-5.2 over-clarifies in 38% of trials with questions, while Gemini models under-clarify in 31%.Across models, under-clarification rises from 26% to 48% among trials where the task fails despite questions, pointing to question quality as a potential recovery bottleneck.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 Agentic Prompting Strategies

• CodeAct performs best on overall Pass@3, while Reflexion and Plan & Execute do better on checkpoint accuracy and outcome-critical tasks; Plan & Execute asks questions in 80% of trials.On outcome-critical tasks with the user reintroduced, Reflexion and Plan & Execute reach 17.9% and 18.4%, respectively, with a 4% checkpoint-accuracy improvement over ReAct and CodeAct.Source: §4.5 · p. 9

Ambiguity Clarification Benchmarks

• ClariQ [1], AmbigQA [8], QuestBench [7], UserBench, and ClarifyBench [12] study ambiguity or clarification in short-context scenarios; LHAW extends evaluation to cost-sensitive long-horizon workflows.UserBench and ClarifyBench simulate users, while QuestBench tests whether models ask the right question in constraint-satisfaction tasks.Source: §5 · p. 10

Interactive Ambiguity Resolution

• Zhang and Choi decompose clarification into when, what, and how to ask, while LHAW generates underspecified variants to study agent behavior under missing information.Their IntentSim approach reports doubled performance gains by clarifying 10% of ambiguous cases, in NLP tasks such as QA, MT, and NLI.Source: §5 · p. 10

Long-Horizon Agent Benchmarks

• THEAGENTCOMPANY, SWE-Bench [5], and τ-bench [18] evaluate long-horizon or tool-agent tasks but assume well-specified tasks; LHAW introduces controlled underspecification.THEAGENTCOMPANY provides enterprise simulation, SWE-Bench evaluates code repair, and τ-bench tests tool-agent-user interaction.Source: §5 · p. 10

Synthetic Data for Evaluation

• Odyssey-Bench [14] generates long-horizon workflow tasks but emphasizes well-defined tasks, while Pan et al. [9] create interactive coding tasks and measure steerability; LHAW targets outcome-critical underspecification.LHAW variants focus on missing information that causes task failure rather than only degraded quality.Source: §5 · p. 10

Implications for Agent Development

• Current agents lack robust uncertainty quantification, strategic clarification that weighs information value against interruption cost, and measurement of behavior under underspecification.Observed rollout behaviors include cutting corners, removing restrictions, and becoming frustrated with users.Source: §6 · p. 10

Limitations

• LHAW operates at the prompt level and addresses underspecification, not semantic ambiguity from conflicting context or multiple valid interpretations.Agents may infer missing information through environmental exploration; repeated sampling and terminal-state analysis classify inferable ambiguity as benign.Source: §6 · p. 10

Future Work

• Future work includes environment-level ambiguity, human-curated validation, model post-training, and explicit user-time or interruption costs in a decision-theoretic value-of-information framework.The current analysis measures clarification efficiency as performance gain per question, without explicitly quantifying interaction cost.Source: §6 · p. 10

7. Conclusion

• LHAW is a synthetic pipeline for controllable underspecification in long-horizon autonomous workflows.The authors report validating the pipeline across several agent benchmarks and demonstrating controllable generation to study ambiguity types.Source: §7 · p. 10
• The paper analyzes information costs and clarification failure modes when ambiguity and user interaction are introduced.These analyses accompany the pipeline’s validation and benchmark-based study of ambiguity types.Source: §7 · p. 10
• The authors frame LHAW’s evaluation goal as measuring whether agents succeed, know when to ask for help, and how much help they need.The conclusion presents this as a foundation for future agent evaluation.Source: §7 · p. 10

Impact Statement

• The impact statement links appropriate clarification-seeking with avoiding silent failures, while noting that increased agent permissions may risk harmful behavior.Source: Impact Statement · p. 11

A.1 Performance by Information Dimension

• Tables 6 and 7 report Pass@3 and average checkpoint progress by information dimension, with gains from the user tool shown in parentheses.The tables mark the TAC Context dimension as having insufficient samples after filtering.Source: Table 6 · p. 13, Table 7 · p. 13, §A.1 · p. 13

A.2 Performance by Ambiguity Class across Benchmarks

• Performance across ambiguity types is reported for MCP-Atlas, TheAgentCompany, and SWE-Bench Pro; the MCP-Atlas table notes near-complete failure on outcome-critical variants.Source: §A.2 · p. 13, Table 8 · p. 14

A.3 Ablation: Agentic Prompting Strategies

• The GPT-5.2 TheAgentCompany ablation compares prompting strategies that ask agents to reason, plan, or reflect.Full results are identified as Table 11, but the supplied passage does not state their outcomes.Source: §A.3 · p. 14

A.4 Validating Outcome-Critical Candidate Generation

• The pipeline validation evaluates whether severity strategies generate outcome-critical ambiguities under the stated filtering criteria.The supplied passage points to Table 12 but does not report its outcome distribution.Source: §A.4 · p. 14

A.5 Critical Blocker Property

• Outcome-critical variants show near-complete failure without clarification.Source: §A.5 · p. 14

A.6 Cross-Benchmark Generalization

• Aggregated Pass@3 (%) across benchmarks is reported as a model average.The supplied passages identify Table 14 but do not provide its values.Source: §A.6 · p. 14, Table 14 · p. 17
• The ambiguity-class tables cover TheAgentCompany and SWE-Bench Pro, with outcome-critical variants described as causing near-complete failure on TheAgentCompany.Source: Table 9 · p. 15, Table 10 · p. 15
• Table 11 describes a GPT-5.2 TheAgentCompany prompting ablation, reporting Pass@3, checkpoint progress, question-asking rate, and questions per trajectory.Parenthetical values indicate improvement with the user tool.Source: Table 11 · p. 16
• Table 12 evaluates how severity strategies affect outcome distributions on TheAgentCompany candidates using Gemini-3-Flash.The caption specifies 233 candidates.Source: Table 12 · p. 16
• Outcome-critical variants show near-complete failure without clarification, as summarized by the fidelity table.Source: Table 13 · p. 16

B. Benchmark Dataset Distribution

• The 285-variant benchmark maps MCP-ATLAS variants 1:1 to tasks, versus 85 variants from 13 tasks in THEAGENTCOMPANY and 100 from 75 in SWE-BENCH PRO.MCP-ATLAS skews toward outcome-critical variants (50%), while SWE-BENCH PRO is more balanced across classes. Goal is the most prevalent dimension overall; multi-segment removals in MCP-ATLAS count each segment, yielding 162 dimension instances from 100 variants, and THEAGENTCOMPANY has no context-type underspecification.Source: Figure 3 · p. 18, B. Benchmark Dataset Distribution · p. 18

F.3 User Simulator Prompt for MCP-ATLAS

• The simulator may provide omitted details only when they appear in the original task and must not guess or answer from information outside it.For compound questions, it provides the details it knows and says when it lacks the rest.Source: §F.3 · p. 26

F.4 Agent System Prompt

• The agent prompt emphasizes exact answers and makes ask_user available, but leaves the decision to clarify to the agent.It signals that precision matters without requiring a question when information is missing.Source: §F.4 · p. 27

G. Evaluation Harness Implementation

• The harness section introduces how each benchmark’s evaluation infrastructure was adapted for underspecification experiments.Source: G. Evaluation Harness Implementation · p. 27
• TheAgentCompany, SWE-Bench Pro, and MCP-Atlas experiments use their respective benchmark harnesses to execute and score tasks.TheAgentCompany uses a Docker-based OpenHands harness with the ask_user tool; SWE-Bench Pro applies patches and runs test suites; MCP-Atlas scores checkpoint completion through its tool harness.Source: G. Evaluation Harness Implementation · p. 27
• Experiments use official API endpoints for five frontier models and run three independent trials per variant for Pass@3.Source: G. Evaluation Harness Implementation · p. 27

H. Value of Information: Additional Details

• Figure 4 plots overall Pass@3 and average checkpoint performance against gain per user question; its upper-right quadrant marks agents with high task success and high value per question.Source: H. Value of Information: Additional… · p. 27, Figure 4 · p. 28

I.1 Prompt for Judging Ask User into Taxonomy

• The judge taxonomy distinguishes question quality, targeting, information integration, over- and under-clarification, timing and strategy, and response-misinterpretation failures.Its categories include poorly formulated questions, missed critical information, failure to use correct responses, and excessive or insufficient questions.Source: §I.1 · p. 28
• Flags use explicit prerequisites and YES/NO/N/A judgments; missed-critical-segment flags require criticality >= 0.5 and guessability <= 0.5.Other operational criteria include whether an agent uses correct user responses and whether it repeats questions.Source: §I.1 · pp. 28–30

I.2 Ask User Failure Modes in Failed Trials

• Under-clarification rises from 26% to 48% on average, and question targeting errors from 41% to 58% in trials that fail despite questions.The analysis identifies question quality as a bottleneck for recovering baseline performance.Source: Table 17 · p. 31, §I.2 · p. 31

I.3 Sub-Taxonomy Ask User Failures for GPT-5.2

• Compound questions and missed critical segments dominate across ambiguity classes, with outcome-critical variants showing the highest under-clarification rates.The figure reports GPT-5.2 failure modes across the full sub-taxonomy.Source: Figure 5 · p. 32, §I.3 · p. 32

J. User Persona Details

• Clarification frequency ranges from 0.99 calls/trial for outcome-critical tasks under the Supervisor persona to 0.22 calls/trial for benign tasks under the Executive persona.The analysis reports a gradient across task ambiguity classes and user personas on MCP-ATLAS with Sonnet-4.5.Source: J. User Persona Details · p. 32, Figure 6 · p. 33