AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

• AM-Bench adapts 48 tasks into Full references and six ambiguous or missing goal, context, and constraint variants, evaluating five models and four harnesses over 20,160 runs.Completion and Compliance are assessed separately, and Success requires both.Source: §1 · p. 2
• Across information gaps, goal uncertainty impedes Completion, context uncertainty raises token use, and constraint uncertainty weakens Compliance while largely preserving Completion.Missingness yields lower Success than ambiguity in all three domains.Source: §1 · p. 2
• Information gaps change system comparisons: Full-condition advantages may not persist, and rankings vary by model–harness pairing and whether evaluation measures Completion, Success, or repeat consistency.The same harness change can improve one model’s Success while reducing another’s.Source: §1 · p. 2
• Matched Full-run estimates suggest clarification could recover unfinished work, but offers limited relief for constraint violations.Larger estimated gains under missingness narrow the Success disadvantage for context and reverse it for goals, while leaving the constraint gap largely unchanged.Source: §1 · p. 3
• The paper recommends making goals and constraints explicit, supplying execution context, and evaluating model–harness systems across information conditions.It also recommends targeted clarification before actions that depend on unresolved requirements.Source: §1 · p. 3

• Workflow benchmarks range from API and multi-turn tool use to stateful applications and long-horizon command-line tasks; Harness-Bench crosses models and harnesses on shared tasks (Yao et al., 2026).AM-Bench extends that system-level comparison to fully specified and underspecified task conditions.Source: §2 · p. 3
• Risk benchmarks study malicious requests, instruction injection, or underspecified tasks; AM-Bench focuses on benign goal, context, and constraint gaps and separately evaluates completion and compliance.ToolEmu studies underspecified instructions and constraints, while UnderSpecBench varies intent clarity, target certainty, and action scope (Ji et al., 2026).Source: §2 · p. 3
• Clarification research spans language, tool-call, and workflow benchmarks; AM-Bench compares requests across information conditions using matched Full runs as a common reference.This comparison avoids relying on a particular user or user-simulator response strategy.Source: §2 · p. 3

• The benchmark groups fully specified and underspecified conditions of the same task and provides a common execution interface for model–harness comparisons.Clarification opportunities are assessed using explicit requests and matched Full runs from the same system.Source: §3 · pp. 3–4
• Each task has goal, context, and constraint information; ambiguity permits multiple reasonable interpretations, while missingness omits a key requirement.Combining these forms across the three domains yields six variants, with Full supplying all three types of information.Source: §3 · p. 4, Table 1 · p. 4
• Completion checks required outputs, final state, and workflow actions; Compliance checks constraints, and Success requires both checks to pass.Clarification requests are identified from traces; turns and tokens measure cost, including for unsuccessful runs.Source: §3 · p. 4, Figure 2 · p. 5
• Clarification is an explicit request for task-relevant information or permission; matched Full runs provide a common complete-information reference.The estimated gain assumes complete information restoration and Full-like execution; more requests need not yield larger gains.Source: §3 · p. 4

• AM-Bench adapts 48 practical, externally executable tasks: 28 from Terminal-Bench 2.0 and 20 from Harness-Bench.Selected workflows have checkable outcomes and include directory permissions, audio transcription, and access-log reports.Source: §4 · p. 5
• Each case combines a request, initial workspace, task-specific MCP server, and evaluator, with core, distractor, and restricted tools.Distractor tools offer alternative goals, objects, or sources; restricted tools access protected information or act outside task constraints.Source: §4 · p. 5
• Each adapted task receives a Full condition and six variants targeting information in requests, guidance, tool descriptions, or tool returns.GPT-5.6-Sol assisted in drafting variants, which the authors reviewed; the benchmark contains 48 Full references and 288 underspecified instances.Source: §4 · p. 5
• All harnesses use a common interface to each case’s MCP server and workspace, allowing configurations to execute the same cases.Run records link traces to artifacts, action logs, turns, and token counts.Source: §4 · pp. 5–6
• Task evaluators check artifacts, state, and required actions for Completion, and action logs for Compliance.In the ACL example, producing the output alone is insufficient, and accessing a prohibited profile or credential source fails Compliance even if permissions are correct.Source: §4 · p. 6
• GPT-5.5 analyzed and summarized execution records and identified clarification requests; authors audited run-level labels against the records.Clarification estimates match Full runs by task, model, and harness.Source: §4 · p. 6

5.1 EXPERIMENTAL SETUP

• The study crosses five models with four harnesses across the same tasks and conditions, using three runs per setting for 20,160 total records.The setup uses a 300-second timeout and 2,880 records per condition.Source: §5.1 · p. 6

5.2 RQ1: EFFECTS OF TASK UNDERSPECIFICATION

• Goal underspecification puts Completion 57.2 pp below Full, context adds 66.6k tokens without improving Success, and constraints cut Compliance 73.0 pp while largely preserving Completion.These domain averages combine ambiguous and missing conditions.Source: §5.2 · p. 6, Table 2 · p. 7
• Within every domain, missingness lowers Success than ambiguity, even when it raises Completion or reduces token use.Constraint–Missing raises Completion by 5.2 pp over Constraint–Ambiguous, yet every completed run fails the reference constraint checks; Goal- and Context–Missing use 7.2k and 15.1k fewer tokens than their ambiguous counterparts while Completion falls.Source: §5.2 · pp. 6–7

5.3 RQ2: EXECUTION ACROSS MODELS AND HARNESSES

• Completion, Success, and repeat consistency can rank systems differently: Codex succeeds in 8.0 pp more groups at least once than NullClaw, but in 2.4 pp fewer groups across all three runs.Under Constraint–Missing, every system has zero Success despite varied Completion changes; under Constraint–Ambiguous, DS-Flash’s 5.0 pp Completion lead over GLM-5.3 becomes a 12.7 pp Success deficit.Source: §5.3 · pp. 7–8, Table 3 · p. 8
• Full-condition advantages do not reliably persist: GPT-Sol and GPT-Luna fall from leading Full Success to fourth and fifth on the underspecified mean, while DS-Pro rises from last to first under Goal–Missing.Harness rankings also shift by condition, and switching OpenClaw to NullClaw raises DS-Pro’s mean Success by 8.6 pp but lowers GPT-Sol’s by 8.1 pp.Source: §5.3 · p. 8

5.4 RQ3: CLARIFICATION OPPORTUNITIES

• Matched-Full clarification estimates raise Completion and Success most for goals, improve context outcomes with lower token use, and leave constraint Compliance at only 30.1%.Estimated Success gains are 15.3 pp for goals, 8.1 pp for context, and 3.1–3.2 pp for constraints; constraint Compliance improves by 4.5 pp.Source: Table 2 · p. 7, §5.4 · p. 9
• Estimated clarification gains reverse missingness’s 12.5 pp Success deficit for goals, but leave deficits of 5.3 pp for context and 33.6 pp for constraints.Across six-condition averages, clarification-rate and estimated-gain SDs are larger across harnesses than models: 7.4 versus 3.9 pp and 4.3 versus 2.2 pp, respectively.Source: Figure 4 · p. 8, §5.4 · p. 9

6 CONCLUSION AND DISCUSSION

• AM-Bench finds that information gaps alter execution and system comparisons, while estimated clarification gains address only part of these deficits.The conclusion calls for clearer requirements, evaluation across information conditions, and clarification support during execution.Source: §6 · p. 9
• Users should state permitted operations and data access first, then specify the required result and the inputs needed for completion.The paper recommends making constraints and goals explicit and supplying execution context.Source: §6 · p. 9
• Model developers should compare Completion, Success, and repeat consistency across Full and underspecified conditions; harness developers should preserve requirements and support clarification before dependent actions.Harness changes should be tested with each supported model because the same change can help one model and hinder another.Source: §6 · pp. 9–10

AI USE STATEMENT

• GPT-5.6 Sol helped generate synthetic task variants, GPT-5.5 identified clarification requests in traces, and GPT-6 Astra polished the manuscript; authors reviewed and verified all AI-assisted outputs.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The benchmark adapts tasks from Terminal-Bench 2.0 and Harness-Bench, documenting their sources in the paper.Cases involving protected information or restricted actions assess adherence to task-specific constraints.Source: ETHICS STATEMENT · p. 10

• The supplementary package includes benchmark source code and normalized run records.Source: REPRODUCIBILITY STATEMENT · p. 10
• Selected execution evidence and scripts for reproducing reported statistics and figures are also provided.Source: REPRODUCIBILITY STATEMENT · p. 10
• The AM-BENCH project repository is available at the stated GitHub address.Source: REPRODUCIBILITY STATEMENT · p. 10

A COMPARISON WITH RELATED BENCHMARKS

• Prior benchmarks address long-horizon execution, model–harness comparisons, underspecification risks, and clarification or interaction in tool tasks.Examples include Terminal-Bench 2.0 and Harness-Bench for execution; ToolEmu and UnderSpecBench for risks; and ClarifyBench, AppWorld-UL, and LHAW for interaction.Source: A COMPARISON WITH RELATED BENCHMARKS · p. 13
• AM-BENCH combines grouped information conditions, a complete model × harness comparison, and evaluation of execution and Full-referenced clarification opportunities.Source: A COMPARISON WITH RELATED BENCHMARKS · p. 13, Table 4 · p. 13

B BENCHMARK CONSTRUCTION

• The construction process adapts Full tasks into underspecified variants and links the resulting artifacts and action records to Completion, Compliance, and Success.Source: B BENCHMARK CONSTRUCTION · p. 13

B.1 TASK COVERAGE

• The benchmark contains 48 adapted tasks: 28 from Terminal-Bench and 20 from Harness-Bench, assigned to primary workflow classes.Tables 5 and 6 present the workflow categories and task-source mapping.Source: §B.1 · p. 14, Table 5 · p. 14, Table 6 · p. 14

B.2 TASK CONSTRUCTION

• A GPT-5.6-Sol skill drafts three ambiguous and three missing variants per Full task, with authors reviewing the generated variants.The instructions weaken one information domain at a time while keeping other information close to Full and checking task, tool, and workspace changes.Source: §B.2 · p. 15

B.3 TASK EXAMPLES

• The examples vary goal specificity, operational context, and constraint clarity while retaining distinct task elements across conditions.The file-summary variants alter the deliverable while retaining the input path; sales-rollup variants alter source guidance; and meeting-scheduling variants change restrictions and tool descriptions.Source: §B.3 · pp. 16, 18, 19

B.4 TASK EVALUATION

• Completion and Compliance are scored separately, so a run can complete the workflow yet fail Success after prohibited access.The recorded ACL example completed the workflow and artifact but included prohibited calls, yielding T = 1, S = 0, and J = 0.Source: §B.4 · p. 19

C EVALUATION METHODS

• The evaluation-method appendix connects execution settings and aggregated outcomes to clarification assessment and comparisons across model–harness configurations.Source: C EVALUATION METHODS · p. 20

C.1 EXECUTION SETTINGS

• Runs use isolated containers, a 300-second timeout, and three repeats per task and condition, with each harness connecting to a task-local server.Source: §C.1 · p. 20, Table 7 · p. 20

C.2 AGGREGATION AND REPEAT CONSISTENCY

• Outcome rates include all runs, while aggregation equally weights configurations; repeat-consistency statistics are calculated separately for Success, Completion, and Compliance.For k = 3, pass@3 is 100% and pass3 is 0% for a cell with two successes and one failure.Source: §C.2 · p. 20

C.3 CLARIFICATION ASSESSMENT

• Clarification annotations count explicit outward requests, and Full-matched estimates assume complete information restoration and Full-like execution.The estimates replace outcomes only for runs with a request; they do not measure an answered continuation or carry forward violations before the request.Source: §C.3 · p. 21

C.4 COMPARISONS ACROSS MODELS AND HARNESSES

• Model and harness averages are compared by their population spread, while double-centered residuals summarize variation at the model–harness combination level.Source: §C.4 · pp. 21–22

D RQ1: EFFECTS OF TASK UNDERSPECIFICATION

• The RQ1 analyses compare information domains, contrast ambiguity with missingness by domain, and test whether comparisons persist across repeats, tasks, and timeout handling.Source: D RQ1: EFFECTS OF TASK… · p. 22

D.1 EFFECTS ACROSS INFORMATION DOMAINS

• Completion falls 57.2 pp for goals and 51.3 pp for context versus Full, but only 2.1 pp for constraints; Goal–Missing compliant non-completion reaches 80.0%.Constraint gaps retain the highest Completion, but that does not mean the work satisfies the intended requirements.Source: §D.1 · p. 23
• Compliance remains above 90% for goal and context conditions, but falls to 46.6% for Constraint–Ambiguous and 4.7% for Constraint–Missing; mean Success is 23.6% for context, 18.7% for goals, and 16.8% for constraints.Among completed runs, 77.5% under constraint gaps violate the intended constraints, compared with 5.6% under goal gaps and 8.0% under context gaps.Source: §D.1 · p. 23
• Every underspecified condition uses more tokens on average than Full; context conditions use the most, while additional expenditure does not compensate for lost Success.Context–Ambiguous uses about 69% more tokens on average than Full and 2.78 more turns.Source: §D.1 · p. 23

D.2 AMBIGUITY AND MISSINGNESS

• Constraint–Missing raises Completion by 5.2 pp over Constraint–Ambiguous, but lowers Compliance by 41.9 pp and Success by 33.7 pp; all completed runs violate the reference constraints.Source: §D.2 · p. 24
• Goal–Missing cuts mean cost by 7.2k tokens but lowers Completion and Success by 12.1 and 12.5 pp; Context–Missing lowers all three cost measures while also lowering Completion and Success.For goals, median cost falls 35.9% while P90 cost rises 12.3%; for context, mean, median, and P90 costs all fall.Source: §D.2 · pp. 24–25

D.3 CONSISTENCY OF INFORMATION-CONDITION EFFECTS

• Missingness has lower at-least-once and all-three Success than ambiguity in every domain, and Full leads both measures; Constraint–Missing often completes but never succeeds in the repeated-outcome comparison.Each underspecified condition lowers Success on 45–48 of 48 tasks, and the aggregate pattern persists after excluding timeouts.Source: §D.3 · p. 25

E.1 COMPLETION AND SUCCESS

• Similar Success advantages can reflect different failure profiles across models and harnesses; noncompliant completion does not capture constraint violations in unfinished runs.The comparisons distinguish finished noncompliant work from non-completion, so higher Success need not mean fewer failures of every type.Source: §E.1 · p. 26

E.2 SUCCESS AND REPEAT CONSISTENCY

• Under Constraint–Ambiguous, GPT-Sol and GLM-5.3 both have pass@3 = 49.0%, but pass3 is 25.5% and 31.3%; rankings can also reverse by condition.GLM-5.3 leads DS-Flash on Goal–Ambiguous pass@3, while DS-Flash leads on pass3, reversing their six-condition rankings on both measures.Source: §E.2 · p. 27

E.3 INFORMATION CONDITIONS AND COMPONENT PAIRINGS

• Every model has lower Success under each underspecified condition than under Full, even when its relative rank improves.GPT-Sol and GPT-Luna lead under Full, while DS-Flash and GLM-5.3 lead the six-condition mean.Source: §E.3 · p. 28, Table 19 · p. 29
• Harness performance depends on its paired model: NanoBot has the lowest harness mean, yet GLM-5.3 × NanoBot reaches the highest configuration Success.Switching OpenClaw to NullClaw raises DS-Pro’s mean Success but lowers GPT-Sol’s; pairing departures are not uniformly amplified by underspecification.Source: §E.3 · pp. 28, 30, Table 19 · p. 29, Table 22 · p. 31
• Matched runs show that identical reports can conceal different access and workflow failures, with DS-Pro’s Success change concerning access checks and GPT-Sol’s concerning workflow checks.The contrasting task traces do not establish that the submitted output was lower quality.Source: §E.3 · pp. 28–29, Table 20 · p. 29

F.1 CLARIFICATION AND EXECUTION DEFICITS

• Under goal and context underspecification, estimated Completion and Success gains come almost entirely from recovering unfinished work; noncompliant completion changes by at most 0.2 pp.Under constraint ambiguity and missingness, noncompliant completion falls by only 2.0 and 1.9 pp, respectively.Source: §F.1 · p. 31, Table 23 · p. 31
• Clarification requests do not map directly onto unfinished work: in 112 compliant but incomplete Goal–Missing runs, 55 request clarification and 57 do not.Under constraint missingness, 119 of 134 runs with an exposure label complete the task.Source: §F.1 · pp. 31–32, Table 24 · p. 32

F.2 AMBIGUITY AND MISSINGNESS

• 17.4 pp more estimated gain under Goal–Missing than Goal–Ambiguous changes a 12.5 pp Success disadvantage into a 5.0 pp advantage.Goal–Missing has higher request rate, Recovery, and Coverage than Goal–Ambiguous.Source: §F.2 · p. 32, Table 25 · p. 33
• 3.1 pp more estimated gain under Context–Missing than Context–Ambiguous narrows, but does not close, the Success deficit, which remains 5.3 pp.Context–Missing has higher request rate, Recovery, and Coverage, but those differences offset only part of the original disadvantage.Source: §F.2 · p. 33, Table 25 · p. 33
• Only 4.9% of constraint-missing runs request clarification, covering 4.2% of recoverable failures, so the missing–ambiguous Success gap remains almost unchanged.Recovery within requesting runs is 64.8% under constraint missingness versus 55.4% under constraint ambiguity.Source: §F.2 · p. 33, Table 25 · p. 33

F.3 VARIATION ACROSS MODELS AND HARNESSES

• 7.40 versus 3.85 pp are the six-condition clarification-rate SDs across harnesses and models; excluding NullClaw makes model SD larger in all six conditions.The NullClaw contrast accounts for 98.9% of six-condition harness variance in request rates.Source: §F.3 · pp. 33–34, Table 26 · p. 34
• 70.3% Recovery for GPT-Luna accompanies requests covering only 6.9% of recoverable failures, illustrating that estimated outcomes depend on both requesting and coverage.NullClaw has lower Recovery than Codex but higher request rate and Coverage, and its estimated Success reverses their original ordering.Source: §F.3 · p. 34, Table 27 · p. 35
• 27.2 pp versus 0.4 pp are DS-Pro’s estimated gains with NullClaw and OpenClaw, while GPT-Sol’s gains with those harnesses are 8.5 and 11.9 pp.Condition-level gains include negative entries despite positive mean gains for every configuration.Source: §F.3 · p. 34, Table 28 · p. 35, Table 29 · p. 36

G.3 HARNESS DEVELOPERS

• Harness developers should preserve requirements and enable targeted clarification before actions that depend on unresolved goals, inputs, or permissions.Test across supported models and assess whether questions arrive before dependent actions; these practices are recommendations, not evaluated interventions.Source: G PRACTICAL RECOMMENDATIONS · p. 37, §G.3 · p. 37

G.1 USERS

• Users should specify permitted data sources, operations, and approval boundaries, then state the intended deliverable and identify relevant inputs.The proposed priority is execution boundaries, intended result, then information needed to carry it out.Source: §G.1 · p. 37

G.2 MODEL DEVELOPERS

• Model developers should compare Completion, Compliance, joint Success, and repeat consistency across Full and underspecified conditions.The recommendations call for shared tasks and harnesses and inspection by goal, context, and constraint.Source: §G.2 · p. 37