Source-linked AI summary
FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration
Weihang Ding, Junfei Zhan, Yueting Li, Qirong Guo
TL;DR
Deployment asks whether agents can configure systems whose services connect, become ready, and remain observable, a capability not established by repository tests alone. FDE-Bench evaluates that work with replayed deployment artifacts and gated programmatic checks. Across its task grid, the strongest evaluated model resolves 75.0% of tasks, while readiness is the largest failure stage.
Problem
Repository-test success does not establish whether an agent can configure a running, healthy, monitored deployment.
Source: §1 · p. 1Method
FDE-Bench evaluates deployment-configuration tasks by rebuilding submitted declarative artifacts in pristine environments and applying four gated programmatic check layers.
Source: §1 · p. 1, §3.3 · p. 5Results
The strongest evaluated model resolves 75.0% of tasks, and repair resolution rates exceed greenfield rates for every model.
Source: §1 · p. 2Takeaways & Limitations
Fresh replay and check-layer diagnostics connect deployment outcomes to artifacts that can be inspected and replayed.
Source: §6 · p. 9Seven graded behaviors lack a stated specification clause; the reported gap bound matches episodes to the rubric by task name rather than task version.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.
1 INTRODUCTION
FDE-Bench evaluates deployment configuration rather than repository-test performance, asking whether agents can produce a running, healthy, monitored system. It pairs replay-graded tasks with release controls and evaluates model outcomes and failure stages.
1 INTRODUCTION
- Repository tests alone do not establish deployment competence, which also requires services to connect, become healthy, and remain monitored.FDE-Bench evaluates agents on producing a running, healthy, and monitored system.Source: §1 · p. 1
- FDE-Bench targets day 0 and day 1 deployment configuration, complementing held-out evaluation as environment-setup benchmarks are mined as RL training corpora.Submitted artifacts are independently rebuilt and checked against a deployment specification.Source: §1 · p. 1
- The benchmark contains 136 tasks from a 149-card hand-authored blueprint corpus, with a four-arm release gate defining acceptance criteria.Tasks pair natural-language deployment tickets with a machine-readable specification of customer-observable requirements.Source: §1 · pp. 1–2
- Tasks span greenfield configuration and repair, while replay rebuilds submitted declarative artifacts in a pristine sandbox for hidden checks.Four gated binary programmatic layers prevent downstream credit after failed builds or crash-looping pods; scoring uses no LLM judge.Source: §1 · p. 2
- The best evaluated agent resolves 75% of tasks; mean repair resolution rate is 30.7 percentage points higher, with a positive gap for all seven models.No zero-intelligence strategy resolves any evaluated task; check-layer and termination-status breakdowns localize deployment failures.Source: §1 · p. 2
- The contributions include a shared-scaffold evaluation of seven models against three zero-intelligence floors and a reference skyline, plus failure attribution and validity analyses.The protocol reports macro-averaged resolution rate with Wilson intervals and includes specification-leak, gameability, sandbox-isolation, and contamination analyses.Source: §1 · p. 2
2 RELATED WORK
FDE-Bench differs from benchmarks centered on developer-environment setup, code changes, or day 2 operations by evaluating deployment configuration through replayed system behavior and specification conformance.
Development-environment setup
- Development-environment benchmarks primarily assess setup or whether repository tests run, whereas FDE-Bench makes replica conformance, connectivity, probe semantics, and alerting explicit requirements.The compared setup benchmarks validate outcomes such as imports, compilation, dependency installation, or repository tests.Source: §2 · p. 2
Deployment and operations
- FDE-Bench tests day 0 deployment configuration for fixed applications through fresh replay, unlike day 2 operations benchmarks that start with systems already deployed.DeployBench instead grades a one-shot batch outcome, while NetArena generates scenarios on demand and FDE-Bench rotates held-out tasks in waves.Source: §2 · p. 2
Repository-level software and ML engineering
- Unlike repository-level engineering benchmarks, FDE-Bench holds application code fixed and evaluates deployment configuration on the system reconstructed from submitted artifacts.Repository-level benchmarks center evaluation on code changes.Source: §2 · p. 2
Terminal and OS generalists
- Terminal and OS benchmarks cover broad interactive tasks, while FDE-Bench holds its seven-model baseline interface fixed and measures deployment outcomes through replay and gated checks.Figure 1 depicts task construction and agent inputs, artifact submission, and fresh-grader rebuilding over pristine fixtures.Source: §2 · p. 3, Figure 1 · p. 3
3 THE FDE-BENCH BENCHMARK
FDE-Bench evaluates declarative deployment artifacts for fixed applications by replaying them in a fresh environment and applying gated checks. Its tasks cover configuration and repair scenarios, while construction and execution controls address solvability, reproducibility, and grading scope.
3 THE FDE-BENCH BENCHMARK
- The benchmark contains 136 tasks from a 149-card hand-authored blueprint corpus, requiring agents to build images, author or repair orchestration manifests, and configure health monitoring.Tasks submit declarative artifacts for fresh redeployment.Source: §3 · p. 3
3.1 TASK DEFINITION
- Each task combines a customer-facing ticket, a specification of observable requirements, a workspace, a pinned environment, hidden checks, and a reference solution.The specification covers services, replicas, ports, endpoints, health semantics, monitoring, and global constraints; agents see the ticket, specification, and workspace, but not the checks or reference solution.Source: §3.1 · p. 3
3.2 DEPLOYMENT SCENARIOS
- Scenarios span containerization, orchestration, health monitoring, and a full-pipeline family; repair tasks mutate validated configurations with incident-grounded faults that break required checks.Greenfield tasks start from source and specification alone, while repair tasks start from a complete but deliberately broken environment.Source: §3.2 · p. 4
- Task applications are original and modeled on production service patterns, with health-semantics quirks intended to defeat copy-pasted probe defaults.Blueprint cards draw on real community incidents, and a four-arm release gate requires the reference solution to resolve while do-nothing, specification-transcription, and generic-stub arms fail.Source: §3.2 · p. 4
- A public development split and rotating held-out waves use unreleased blueprint-corpus cards, with canary identifiers and logged egress supporting contamination monitoring.Task identity is tracked by card id, and results are not compared across waves.Source: §3.2 · p. 4
3.3 SANDBOX EVALUATION PROTOCOL
- At submission, replay collects declared graded artifacts, tears down the live environment, reapplies the artifacts over pristine fixtures, and runs hidden checks.Edits outside graded artifact globs and imperative live state are discarded rather than included in the replayed submission.Source: §3.3 · p. 5
- Four gated layers check build, readiness, behavior, and specification conformance, with build and readiness gating the downstream checks.Behavior checks include cross-service probes, PromQL series, and alert behavior; conformance checks include replicas, image-size quotas, non-root execution, and secret constraints.Source: §3.3 · p. 5
- Each layer receives weight 0.25, and the binary score lattice measures achieved deployment stages rather than the fraction of correctly authored configuration.A task is resolved only when all four layers pass; timing assertions use stability windows rather than instantaneous timeouts, and scoring is programmatic rather than LLM-judged.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments evaluate seven models under a shared scaffold and find substantial variation in task resolution, with repair tasks consistently easier than greenfield tasks. The results also identify orchestration as the weakest macro-averaged axis and show that agents leave ten tasks unresolved.
4.1 SETUP
- The evaluation compares seven models with three zero-intelligence floors and reference-solution and ensemble-union benchmarks.RR intervals are 95% Wilson intervals; pairwise tests use family-clustered sign-flip permutation tests across 14 task families.Source: §4.1 · pp. 5–6
- All seven models use the same four-tool loop and fixed prompt without per-model tuning, and each task-model cell has one episode under its task budget.Provider sampling defaults were used; this single-episode evaluation does not estimate variation across repeated runs.Source: §4.1 · p. 6
4.2 MAIN RESULTS
- 75.0 percent RR and 0.857 DS make Gemini-3.6-Flash the top observed model on the 136-task grid, while Claude-Sonnet-5 ranks second at 72.8 percent RR and 0.855 DS.The lead over Flash-Lite is supported (p = 0.0015), but the lead over Claude-Sonnet-5 is not (p = 0.59).Source: §4.2 · p. 6, Table 1 · p. 7
- 82.0 percent repair RR versus 51.3 percent greenfield RR yields a 30.7-point gap with the same sign for all seven models.The do-nothing floor starts repair workspaces at 0.418 mean DS versus 0.004 for greenfield, so raw repair DS includes a built-in floor that RR does not.Source: §4.2 · pp. 6–7
- 65.2 percent macro-averaged RR on orchestration is below containerization at 67.1 percent and health monitoring at 73.1 percent.The ten tasks unresolved by every model are all greenfield, concentrated in containerization, with seven single-image tasks and three Kubernetes tasks.Source: §4.2 · p. 7
- 92.6 percent ensemble-union RR is 17.6 percentage points above the best single agent, while ten tasks remain unresolved by all seven models.Reference solutions resolve all 136 tasks; the strongest agent's 75.0 percent RR also exceeds the release calibration target of 35–60 percent.Source: §4.2 · p. 7
5 ANALYSIS
Across the analysis, readiness is the largest failure stage, and every evaluated model performs better on repair than on greenfield tasks. Replay diagnostics also identify cap-ended episodes, task-disclosure and rubric issues, and a small human comparison.
5.2 GREENFIELD VERSUS REPAIR
- 20.9 to 46.1 percentage points separate repair from greenfield RR across models, with DeepSeek-V4-Flash showing the largest gap.The groups contain 70 repair and 66 greenfield tasks with shared scenario families and specification style; mode-specific rankings differ from overall RR.Source: Figure 2 · p. 8, §5.2 · p. 8
5.1 FAILURE-STAGE BREAKDOWN
- 110 of 313 unresolved episodes fail at readiness, the largest failure state; 12.2 percent of build-passing episodes do not reach a stable Ready rollout.The failure mix varies by model: Claude-Sonnet-5 fails most at conformance and DeepSeek-V4-Flash at behavior.Source: §5.1 · p. 8
5.2 GREENFIELD VERSUS REPAIR
- GPT-5.6-Terra and Claude-Sonnet-5 resolve 95 and 99 tasks at median step counts of 9 and 29, respectively.DeepSeek-V4-Pro and DeepSeek-V4-Flash use medians of 35 and 30 calls; four models hit a step or wall-clock limit in 42 to 61 episodes.Source: §5.2 · p. 8
5.3 FAILURE TAXONOMY AND VALIDITY
- Replay assigns all 313 unresolved episodes to build (52), readiness (110), behavior (97), or conformance (54); caps ended 103, including 38 that passed build and readiness.Failure stages follow a reporting priority; behavior-fail includes episodes that pass conformance, while the Deployment Score retains parallel behavior and conformance branches.Source: §5.1 · p. 7, §5.3 · p. 8
- An early specification leak exposed 58 of 61 hidden check constants verbatim, and a zero-intelligence transcriber solved pilot tasks, voiding those results.The agent-visible files were scrubbed, and specification-transcription and generic-stub submissions became permanent release-gate arms.Source: §5.3 · p. 8
- Stability windows reduce sensitivity to transient conditions but do not quantify flakiness; repeated replay at scale remains future work.The contamination controls include corpus-wide canary GUIDs and per-episode egress-proxy access logs.Source: §5.3 · p. 9
- A probe exposed a disclosure defect: verification text omitted a registry pull after image build, while three additional probe tasks resolved in valid episodes.One task’s replay failed at the omitted pull stage; another yielded artifacts scoring 1.0 on replay despite an API error and was retained only as an artifact-level diagnostic.Source: §5.3 · p. 9
- On the 25-task subset, the engineer-assisted Claude-Sonnet-5 resolved 23 tasks versus 18 for autonomous Claude-Sonnet-5 on the same tasks.The comparison has one participant, unlogged session timing, and a confound between human judgment and the richer scaffold.Source: §5.3 · p. 9
6 LIMITATIONS AND CONCLUSION
The conclusion presents replay and check-layer diagnostics as a basis for examining deployment outcomes, while noting limits in repeated-run coverage, adversarial testing, and specification alignment.
6 LIMITATIONS AND CONCLUSION
- One episode per model–task cell means Wilson intervals describe variation across tasks, while timed checks remain sensitive to host load.The reference archive includes 67 runs and 1,138 check observations across 30 development tasks, but complete oracle-artifact and shared-grader identities were not retained.Source: §6 · p. 9
- Three author-designed attacks test corresponding checks, but broader attack coverage requires independently designed submissions.Source: §6 · p. 9
- Seven graded behaviors lack stated specification clauses across three Kubernetes tasks and one Compose task; v1.1 closed 56 of 63 declared gaps.The gap bound matches episodes by task name rather than version, and specialized interfaces may change model results under the same task and grading protocol.Source: §6 · p. 9
- The strongest evaluated model resolves 75.0 percent of tasks, while ten remain unresolved by all seven models; readiness accounts for 110 of 313 failures.Mean repair RR is 30.7 percentage points higher than greenfield; fresh replay connects outcomes to reproducible deployment artifacts.Source: §6 · p. 9
REPRODUCIBILITY STATEMENT
The reproducibility statement archives task versions, score records, evaluation infrastructure, and version-specific task and grader evidence.
REPRODUCIBILITY STATEMENT
- The repository archives v1.0.0-rc1 with an MIT license, 136 tasks, the harness and grader, release-gate verdicts, and protocol documentation.v1.1 retains the v1.0 records and adds revised specifications, corrected workload discovery, re-gating ledgers, and 225 fresh episodes.Source: REPRODUCIBILITY STATEMENT · p. 10
- The revised DS uses equal layer weights, while original scores using 0.20/0.25/0.35/0.20 remain archived; both require all four layers for resolution.Each model–task cell has one episode with harness seed 900, and the protocol forbids selecting a better score by rerunning.Source: REPRODUCIBILITY STATEMENT · p. 10
- The server sweep included a 192-core host alongside earlier 16-thread laptop episodes, and two release-gate arms passed under memory pressure only on an idle host.The appendix records execution settings and missing host metadata; timed checks used a dedicated idle host.Source: REPRODUCIBILITY STATEMENT · p. 10
- Task and grader versions retain separate results; v1.1 fixed three invalid-task defects, while the under-described registry-pull task remained in the grid.Three current-version reference replays had matching outcomes across all 13 checks; subsequent Harbor parity tests have separate records.Source: REPRODUCIBILITY STATEMENT · p. 10
AI USE STATEMENT
The statement describes generative AI assistance in research and manuscript work, while distinguishing it from model-generated agent actions and programmatic evaluation.
AI USE STATEMENT
- Generative AI tools assisted with research design, code implementation and debugging, log analysis, literature work, manuscript preparation, and figure preparation.Source: AI USE STATEMENT · p. 13
- Evaluated language models generated the agent actions and deployment artifacts studied in the experiments.Source: AI USE STATEMENT · p. 13
- The replay evaluator computed scores using programmatic checks without an LLM judge.Source: AI USE STATEMENT · p. 13
- The authors reviewed the text, checked claims against methods and evidence, and inspected and ran the experimental and analysis code.Source: AI USE STATEMENT · p. 13
A WORKED TASK EXAMPLE
The worked example turns a deployment specification into observable service obligations while leaving implementation choices to the agent.
A WORKED TASK EXAMPLE
- The example is a condensed specification for a three-service Compose task in the full-pipeline family.The preamble lists omitted implementation choices, while contract numbers restrict which constants hidden assertions may share with agent-visible text.Source: A WORKED TASK EXAMPLE · p. 14
- The three-service contract checks published access, readiness, write-then-read behavior, credential rejection, durable catalogue storage, and queryable request metrics.Hidden checks generate requests at grading time to test behavior beyond fixed responses.Source: A WORKED TASK EXAMPLE · p. 14
- The agent must build both images from source, publish declared ports, apply the schema before traffic, deliver credentials without plaintext, and configure scraping.Base image, server, worker count, and probe cadence remain implementation choices.Source: A WORKED TASK EXAMPLE · p. 14
B BENCHMARK CARD
The benchmark evaluates deployment-configuration competence under a frozen, isolated protocol, with task files defining the contract and replay artifacts. Its card also states intended uses, scope boundaries, stewardship policies, and known limitations.
B BENCHMARK CARD
- FDE-Bench measures whether agents can produce Docker, Compose, or Kubernetes artifacts that satisfy deployment specifications, using programmatic checks and no LLM judge.Reported baselines use seven models, a shared four-tool scaffold and fixed prompt, allow-listed registry egress, and fixed budgets without per-model tuning.Source: B BENCHMARK CARD · pp. 14–15
- The benchmark supports system comparison, progress tracking, failure analysis, and repair-versus-greenfield analysis, with the dev split for development and held-out split for confirmation.Source: B BENCHMARK CARD · p. 15
- Scores do not certify production security or transfer to live multitenant clusters, and cloud-provider infrastructure-as-code is outside the benchmark’s scope.Checks verify specific hardening properties rather than absence of vulnerabilities.Source: B BENCHMARK CARD · p. 15
C TASK SCHEMA AND WORKED EXAMPLE
- Task manifests identify family, mode, and deployment surface; tickets describe the request, while spec.yaml states observable service, behavior, health, monitoring, and constraint requirements.The workspace supplies application code and partial or deliberately broken configuration; the grading contract specifies collected artifacts and replay commands.Source: C TASK SCHEMA AND WORKED EXAMPLE · p. 16
- A second mechanically different reference solution is required alongside the hidden author solution; the frozen evaluation records successful runs on 132 of 136 tasks.The v1.1 gate re-ran it on 64 tasks and it resolved every one.Source: C TASK SCHEMA AND WORKED EXAMPLE · p. 16
D CORPUS COMPOSITION
- Table 4 maps task names to internal IDs and families, while Table 3 gives corpus composition and Figure 3 shows release-gate score-lattice distributions.Source: D CORPUS COMPOSITION · p. 16
E FULL-PIPELINE FAMILY AS A CASE STUDY
The five-task full-pipeline family combines containerization, orchestration, and health monitoring, and is treated as a case study rather than a per-axis aggregate.
E FULL-PIPELINE FAMILY AS A CASE STUDY
- The full-pipeline family has five tasks and is excluded from per-axis aggregates because one task changes resolved rate by 20 percentage points.It is reported as a case study instead.Source: E FULL-PIPELINE FAMILY AS A CASE STUDY · p. 16, Table 3 · p. 17
- Four of five full-pipeline tasks were resolved by at least one agent, but none by all seven; five of seven episodes on one Kubernetes task stalled at readiness.No agent resolved deploy-goods-intake-stack-k8s, which requires an idempotent migration.Source: E FULL-PIPELINE FAMILY AS A CASE STUDY · p. 16
- On these five tasks, do-nothing scored 0.000 throughout, spec-transcription scored above zero on only one, and the generic stub never passed readiness.The results indicate limited partial credit for the tested floor strategies on full-pipeline tasks.Source: E FULL-PIPELINE FAMILY AS A CASE STUDY · p. 17
F RELATED-BENCHMARK COMPARISON
FDE-Bench differs from nearby deployment and DevOps benchmarks in the artifacts and checks it targets. The appendices detail failure accounting, scoring choices, statistical comparisons, and historical probe records.
F RELATED-BENCHMARK COMPARISON
- FDE-Bench grades fixed applications’ Docker, Compose, and Kubernetes artifacts by fresh replay, whereas DevOps-Gym uses containers as an execution environment.DevOps-Gym includes build configuration, monitoring, issue resolution, and test generation; its monitoring stage diagnoses anomalies rather than configuring monitoring.Source: F RELATED-BENCHMARK COMPARISON · p. 17, Table 5 · p. 19
G FAILURE TAXONOMY
- 226 of 952 valid episodes ended at step or wall-clock limits; 103 remained unresolved and 123 resolved on replay.Six token-truncated episodes all failed at build; one episode had unknown termination status.Source: G FAILURE TAXONOMY · pp. 19–20, Table 7 · p. 20
H CHECK-LAYER WEIGHTS AND BINARY SCORING
- Equal 0.25 weights across four binary layers preserve every resolution decision and the overall model Deployment Score ordering on the v1.1 grid.The layers share a score lattice; Resolved Rate is invariant to weights, while Deployment Score depends on intermediate-state credit.Source: H CHECK-LAYER WEIGHTS AND BINARY SCORING · pp. 20–21
I FAMILY-CLUSTERED SIGN-FLIP PERMUTATION TEST
- The family-clustered sign-flip test flips whole scenario families, preserving dependence among related task variants; exact enumeration gives p = 0.0015 and p = 0.59 for the two reported comparisons.The test assumes exchangeability of paired model labels at the family level.Source: I FAMILY-CLUSTERED SIGN-FLIP… · p. 21
J CLAUDE-OPUS-5 PROBE RECORDS
- Across historical selected-task probes, 14 of 15 valid episodes resolved; one API-interrupted attempt had a replay-passing artifact but was excluded under the API-error rule, and two tasks had no probe record.Source: J CLAUDE-OPUS-5 PROBE RECORDS · p. 21
K DISCLOSED TASK DEFECTS AND GRID MEMBERSHIP
The appendix documents task-grid repairs and a remaining specification-disclosure defect, then reports mode-level results and replay/configuration records. These records qualify how task outcomes and reproducibility should be read.
K DISCLOSED TASK DEFECTS AND GRID MEMBERSHIP
- The v1.1 grid retains 136 tasks after repairing three defects and re-running affected tasks; 29 tasks also gained stability-window clauses in their specifications.The re-run replaced 225 cells, leaving 727 frozen and 225 v1.1 cells; recorded costs for 20 episodes were repriced from token counts.Source: K DISCLOSED TASK DEFECTS AND GRID… · p. 22
- Task instrument-ledger-api-busy-metrics remains in the grid despite an under-described replay step; removing it moves every rate by less than one percentage point.Six of seven roster models cleared build and three resolved the task; the omission coincided with build-stage failures for two Anthropic models.Source: K DISCLOSED TASK DEFECTS AND GRID… · pp. 22–23, Table 10 · p. 23
- Seven declared specification gaps remain across four tasks, but no gated layer fails on gap checks alone, so removing those checks changes no resolution or Deployment Score.The two observed episodes failing gap checks also failed 11 checks with stated specification clauses; frozen episodes may have run against different task versions.Source: K DISCLOSED TASK DEFECTS AND GRID… · p. 23
L RESOLVED RATE BY AXIS AND MODE
- The macro-average repair–greenfield Resolved Rate gap is 30.7 percentage points.Table 11 uses denominators of 49, 48, and 34 for the three axes, and 66 and 70 for the two modes.Source: L RESOLVED RATE BY AXIS AND MODE · p. 23, Table 11 · p. 24
M HISTORICAL RUN CONFIGURATION
- Provider records omit sampling settings and dated model snapshots, and the retained episode set spans multiple hosts and incomplete host metadata.Seed 900 identifies the harness episode, not provider sampling; CPU model, full OS/runtime snapshots, and a complete per-episode host map were not archived.Source: M HISTORICAL RUN CONFIGURATION · pp. 24–25
N REFERENCE REPLAY COVERAGE
- Across 90 archived reference replays, each task has identical named check outcomes, while fingerprints match across all three seeds for only seven tasks.A separate three-run replay of containerize-metrics-shim matched hashes and passed all 13 checks each time, with DS 1.0 per run.Source: N REFERENCE REPLAY COVERAGE · p. 25
Related papers
This paper cites
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering2024
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments2024
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents2024
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces2026