PaperlayerView exact revision on arXiv

Source-linked AI summary

FDE-Gym: The Last Mile of AI Delivery

Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang

arXiv:2610.22350v1cs.AIcs.CLcs.SE

TL;DR

Existing agent benchmarks generally assume tasks are already well specified, whereas customer delivery requires agents to identify the underlying problem and discover unstated constraints. FDE-Gym evaluates delivery across industry-adapted customer scenarios using customer business outcomes. In the realistic setting, the best configuration scores 0.56 and resolves 14 of 95 tasks, with losses arising both from constraint discovery and from translating discovered information into effective systems.

  • Problem

    Existing agent benchmarks typically evaluate well-specified tasks, while real-world delivery requires clarifying requirements and uncovering unstated constraints.

    Source: §1 · p. 2

  • Method

    FDE-Gym evaluates agents on 95 tasks from 20 customer scenarios across 18 sectors, using customer environments and business metrics to assess delivery.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    At L3, Claude Opus 5.5 achieves the highest mean score of 0.56 and resolves 14 of 95 tasks; leading models’ losses reflect both constraint discovery and translating facts into effective systems.

    Source: §1 · p. 3, §5.2 · p. 8, §8 · p. 12

  • Takeaways & Limitations

    FDE-Gym makes delivery judgment measurable on customers’ own metrics under shared rules for people and agents.

    Source: §8 · p. 12

    Stakeholders are scripted, unlike less predictable real people, and contact costs are expert-set models of stakeholder time rather than measurements.

    Source: §7 · p. 12

Abstract

from arXiv · show

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

1 Introduction

FDE-Gym targets the gap between completing specified tasks and improving a customer’s business: it tests whether agents can identify the underlying problem, discover constraints, and deliver effective solutions. In its realistic setting, the best configuration scores 0.56 and resolves 14 of 95 tasks, while the benchmark evaluates delivery against customer-defined metrics.

1 Introduction

  • Existing benchmarks generally assume a fixed, well-defined problem, while recent benchmarks address individual aspects of delivery in isolation.The examples include software-engineering, deployment, enterprise, clarification, customer-service, and misleading-request benchmarks.Source: §1 · p. 2
  • FDE-Gym combines 95 tasks across 20 customer scenarios and 18 sectors, testing whether agents identify the underlying problem and improve the customer’s own business metric.A solution addressing the wrong problem earns no credit.Source: §1 · p. 2
  • The FDE-Gym score measures business-metric improvement against a do-nothing baseline of 0 and an expert reference solution of 1, with stakeholder contact costs deducted.Literal compliance with the sponsor’s request scores at most 0.2.Source: §1 · p. 2
  • At L3, Claude Opus 5.5 achieves the highest mean score of 0.56 and resolves 14 of 95 tasks, while five of thirteen configurations resolve none.Selecting the best-performing configuration for each task yields a score of 0.70, with 30 tasks resolved.Source: §1 · p. 3
  • At L3, leading models’ scores fall sharply when they must discover unstated constraints, but obtaining stakeholder facts still does not ensure effective delivery.Leading models score 0.73–0.82 when rules and constraints are provided, versus 0.39–0.62 when they must discover them; on original tasks, L3 scores are at most 0.08 points below L1.Source: §1 · p. 3
  • FDE-Gym introduces a benchmark, a scenario-construction methodology, and an online platform where humans and agents practice delivery under the same grading criteria.The benchmark evaluates whether agents identify the underlying customer problem and deliver a working solution that measurably improves business outcomes.Source: §1 · p. 3

2 Anatomy of a Task

An FDE-Gym task places an agent in a customer engagement with a sponsor and production system, where it must work amid hidden constraints and improve the customer’s metric. The score combines normalized business uplift with incident and stakeholder-contact adjustments.

2 Anatomy of a Task

  • Each task models one customer engagement with one customer, sponsor, and production system to improve.The agent starts with a task instruction, a copy of the system, relevant documents and data, and a staff directory; delivered contents are deployed without modification.Source: §2 · p. 3, §2.1 · pp. 3–4

2.1 Environment and Available Information

  • Agents must infer contact costs, relevant stakeholders, and whether the sponsor’s request is misdirected from documents, data, and system evidence.Available materials can be outdated or incomplete, and legacy code may encode incorrect assumptions.Source: §2.1 · p. 4

2.2 Task Instruction Levels

  • L0 supplies the problem, rules, constraints, and scoring criteria, while L3 presents the sponsor’s original request and requires assessing whether it addresses the underlying problem.The environment and evaluation criteria stay fixed across levels; L0 still leaves solution design and engineering decisions open.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 FDE-Gym Score

  • Normalized business uplift compares the delivered system’s customer metric with the unchanged baseline and expert reference; U = 0 means no improvement and U = 1 matches the reference.U < 0 indicates an outcome worse than the baseline, with all three metrics evaluated on the same held-out data.Source: §2.3 · p. 4
  • The single-run score subtracts stakeholder contact cost from incident-adjusted uplift, and a task is resolved at a score of at least 0.8 without an incident.Contact charges can include messages, trial runs, and days of waiting; the aggregated average floors incident-adjusted uplift at −1 before subtracting contact cost, with equal task weighting.Source: §2.3 · pp. 4–5

3 Building the Benchmark

FDE-Gym converts expert-adapted customer delivery projects into varied, reproducible benchmark tasks with scripted stakeholders and predefined scoring anchors. Its online platform lets people practice under the same workspaces and grading used for agents.

3.1 Scenario Construction from Real-World Delivery Projects

  • Industry experts adapted real delivery projects into 20 scenarios across 18 sectors, yielding 95 tasks that require a working agent, workflow, or service in the customer environment.Projects came from China, Singapore, and the United States; companies and individuals are de-identified or fictional, and some data are adapted from industry sources.Source: §3.1 · p. 5

3.2 Scenario Variants and Underlying Facts

  • Each scenario’s variants alter key facts, so a solution that succeeds on one task may fail on another; no single problem pattern appears in more than one-third of the set.The variants are intended to encourage task-specific evidence evaluation rather than fixed solutions.Source: §3.2 · p. 5, Table 2 · p. 6

3.3 Stakeholder Simulation

  • More than 130 stakeholders use scripted knowledge, misconceptions, disclosure conditions, patience limits, and contact restrictions to make interactions reproducible.Agents must infer stakeholder roles and communication constraints from customer documents; changing the user-simulation model can shift measured success rates by several percentage points.Source: §3.3 · p. 6

3.4 Admission Anchors and Anti-Gaming Validation

  • Admission anchors require the reference to score at least 0.95, the do-nothing baseline to score 0, and literal compliance to score at most 0.2 except when the request is correct and complete.Shortcuts must score at most 0.8; attempts to bypass controls or manipulate grading are capped near zero, and graders were tested against more than 270 such strategies.Source: §3.4 · p. 6

3.5 Expert Review and the Online Platform

  • Industry experts contributed more than 10,000 hours to cross-review and refine tasks, while the online platform gives human participants the same workspaces, stakeholders, and graders as agents.An expert human baseline on L3 tasks is being collected, with results reserved for a future version of the paper.Source: §3.5 · p. 6

4 Experimental Setup

The study evaluates 13 model configurations across instruction levels that vary the information given while holding the environment and grading fixed. It reports task-level performance, costs, and comparisons with external benchmarks, while making the harness and runner open source.

Models

  • The study evaluates 13 model configurations from seven providers using a shared minimal framework with five file and command tools and no model-specific customization.Stakeholder contact is initiated through a command in the workspace.Source: §4 · p. 6

Protocol

  • L3 is the primary setting and covers all 95 tasks; L0–L2 evaluate original tasks, with each model run on 20 tasks at L0, 20 at L1, 17 at L2, and 95 at L3.Each task–instruction-level combination runs three times, ending at delivery or limits of 200 turns, 150 minutes, or USD 8 in model usage cost; deployment evaluates the system state then reached.Source: §4 · pp. 6–7

Reporting

  • Performance reporting includes mean FDE-Gym scores by instruction level, 95% confidence intervals for the full L3 evaluation, normalized business uplift before contact cost, and per-run model usage cost.Table 3 reports scores, confidence intervals, uplift, and contact cost; Figure 5b reports mean usage cost.Source: §4 · p. 7

Comparison with other benchmarks

  • External benchmark success rates are shown alongside FDE-Gym normalized uplift without equating the metrics, using one independent source for consistent model comparisons.Comparable results are unavailable for some cited benchmarks and models; the study selects each provider’s best evaluated model with results available across all three external benchmarks.Source: §4 · p. 7

Availability

  • The task-workspace harness and experiment runner are open source, while the 75 task variants and all reference solutions are held out.The 20 original tasks and their graders are available on request, with three tasks public as examples.Source: §4 · p. 7

5 Results

At L3, the best configuration averages 0.56 and resolves 14 of 95 tasks, while even the best per-task configuration reaches only 0.70. Constraint discovery causes the largest observed score drop, and recovering facts does not ensure effective delivery.

5.1 Even the Best Model Is Far from the Reference

  • At L3, Claude Opus 5.5 leads with a mean score of 0.56, resolves 14 of 95 tasks, and no model resolves more than 15.Selecting the best configuration for each task yields only 0.70 and 30 resolved tasks; every reference solution scores at least 0.95.Source: §5.1 · p. 7, Table 3 · p. 8

5.2 Withholding Constraints Produces the Largest Observed Score Drop

  • 0.28 (95% interval 0.17 to 0.39) is the average score decline from L0 to L1; L1 to L3 adds only 0.04 (−0.03 to 0.14).The L0-to-L1 decline is measured on the same 20 tasks across the three leading models; no model exceeds 0.82 at L0 on those tasks.Source: §5.2 · pp. 7–8

5.3 Recovering Key Facts Does Not Ensure Effective Delivery

  • At L3, leading models obtain 72–85% of key facts and deliver 0.60–0.70 normalized business uplift, yet Opus reaches at least 0.8 uplift on only 10 of 23 all-facts tasks.Across 73 task–model cases where a leading model obtained every key fact, uplift averaged 0.66 and reached at least 0.8 in 27 cases.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Weak Models Frequently Score Below the Do-Nothing Baseline

  • The weakest models score below the do-nothing baseline on 60–83% of tasks, and much of this harm comes from the delivered systems themselves.Before stakeholder contact cost, Mistral Small 4 and Claude Haiku 4.5 (thinking) score below baseline on 51 and 55 tasks, respectively.Source: §5.4 · p. 9

5.5 Model Performance Varies Substantially Across Scenarios and Task Variants

  • On five scenarios no model averages at least 0.5; across task variants, a leading model's median score range is 0.34–0.44 and exceeds 0.5 in 6–8 scenarios.On the two hardest scenarios, news_incident and revenue_dispute, the best scores are 0.18 and 0.27.Source: §5.5 · p. 9

5.6 Higher Spending and Effort Do Not Consistently Correspond to Higher Scores

  • Higher spending and effort do not consistently predict higher scores: GPT-6.1 Sol spends USD 0.24 in 25 turns and scores 0.37, while Gemini 3.8 Flash spends USD 1.54 in 121 turns and scores 0.19.Qwen 3.8 Flash works 169 turns and 110 minutes per run, submits within limits on only 11 of 152 combinations, and scores 0.07.Source: Figure 5 · p. 8, §5.6 · p. 9

5.7 Comparison with other benchmarks

  • Models solving up to 88%, 90%, and 61% of tasks on three component benchmarks score 0.19–0.56 on FDE-Gym; Claude Opus 5.5 resolves 14 of 95 tasks.The benchmarks cover terminal engineering, web-app development from a specification, and financial analyst research; their success rates and FDE-Gym uplift are different quantities.Source: §5.7 · pp. 10–11, Figure 8 · p. 10

6 Related Work

Existing benchmarks generally isolate stages of delivery, while FDE-Gym joins problem interpretation, stakeholder interaction, and business outcomes. Related work informs its task design and reproducibility choices.

Comparison with prior benchmarks

  • Existing benchmarks typically measure one delivery stage; FDE-Gym links stages so misreading the problem can leave good code worthless on the business metric.The comparison organizes prior benchmarks by the capability they isolate.Source: §6 · p. 11
  • τ-bench [9] builds customer-service systems from records, client requirements, and code, while FDE-Gym varies instruction specificity, charges for stakeholder contact, and measures business-metric uplift.τ-bench’s strongest configuration passes 23.9% of simulations versus 82.2% for its expert reference.Source: §6 · p. 11

Task instruction levels

  • XYEval [11] reports relative drops up to 46.7% on XY problems; UnderSpecBench [35] and Implicit Intelligence [23] likewise align with the largest drop occurring when constraints are unstated.FDE-Gym’s leading models lose little when the sponsor’s request is misdirected, possibly because other engagement materials can contradict it.Source: §6 · p. 11

Benchmarks informing the design

  • TheAgentCompany inspired FDE-Gym’s self-contained simulated company, but FDE-Gym substitutes customer engagement and business metrics for colleague-assigned tasks and task completion.LHAW [8] informs missing-information and asking-cost treatment; Vending-Bench [30] motivates pairing an executable business simulation with an engineering deliverable.Source: §6 · pp. 11–12

Methodological design choices

  • FDE-Gym scripts stakeholders because results with LLM-played users depend on the user model, and runs each task three times because single-run success can overstate reliability.Its admission anchors also follow benchmark-auditing practice using do-nothing and shortcut solutions.Source: §6 · p. 12

7 Limitations

The benchmark’s results are bounded by simulated interactions, limited samples and coverage, and the chosen model framework and execution setup.

A simulated customer

  • Scripted stakeholders support reproducibility, but real people may change their minds, answer different questions, or respond differently based on prior interactions.Scenario experts set contact costs as a model of stakeholder time, not a measurement.Source: §7 · p. 12

Sample size across levels

  • L0–L2 cover only 20 original tasks, and three L2 results reuse identical L3 instructions; only the large L0-to-L1 drop is claimed, not further large drops.Differences among L1, L2, and L3 are not large, and the authors claim only that no large further drop is visible.Source: §7 · p. 12

No human baseline yet

  • The reported version has no human baseline, so it does not measure how far human experts can proceed from only the sponsor’s request.The scale is anchored by do-nothing and reference solutions, showing each task is solvable with full knowledge of underlying facts.Source: §7 · p. 12

A single agent framework

  • Results cover models in one minimal agent framework, not their packaged products; a stronger framework may change scores and rankings.Source: §7 · p. 12

Isolation and cost

  • Execution used hosts with varying process-isolation strength; one container per run would offer a stronger guarantee, and metered usage costs are not list-price comparisons.Source: §7 · p. 12

Coverage and contamination

  • The 20 scenarios across 18 sectors are a sample weighted toward regulated operations, and the authors cannot rule out related material in training data.The scenarios derive from real deliveries, though companies are fictional and variants change the facts.Source: §7 · p. 12

8 Conclusion

FDE-Gym evaluates delivery by improvement in the customer’s business, and current models remain far from strong performance on that measure. The conclusion frames closing this gap as a question of delivery judgment.

8 Conclusion

  • The best of 13 configurations averages 0.56 and resolves about one task in seven; leading models lose most when translating facts into effective systems and paying stakeholder contact costs.Weaker models leave customers better served by no delivery on most tasks.Source: §8 · p. 12
  • FDE-Gym makes delivery judgment measurable as one score per task on the customer’s own metric, using the same rules for people and agents.Source: §8 · p. 12

Related papers

Loading 2610.22350v1…