PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman

arXiv:2610.16402v1cs.AIcs.SE

TL;DR

Web-generation benchmarks primarily evaluate single requests or refinement toward a fixed intent, leaving successive development of user-facing applications unaddressed. EvoWeb evaluates cumulative requests with dependency-aware browser tests and tolerance-averaged trajectory scores; experiments find cross-request integration especially challenging.

  • Problem

    Web-generation benchmarks cover single requests or fixed-intent refinement, leaving open how reliably agents extend persistent user-facing applications while integrating earlier functionality and preserving behavior.

    Source: §1 · p. 1

  • Method

    EvoWeb evaluates 320 cumulative requests across 32 applications using dependency-structured features, browser tests of intermediate applications, and trajectory scores integrated across user-tolerance thresholds.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    Cross-request integration is consistently harder than new or earlier-requested functionality; Fable 5 and Kimi K3 lead with Grades of 0.694 and 0.684.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Feature integration and sustained runtime reliability are priorities for improving coding agents in long-horizon, multi-request development.

    Source: §6 · p. 9

    The current benchmark does not include longer trajectories or closed-loop user feedback and repair requests, which future extensions could use to study recovery from mistakes.

    Source: §6 · p. 9

Abstract

from arXiv · show

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.

1 INTRODUCTION

Existing benchmarks do not evaluate user-facing web applications that evolve through successive requests, where new features must integrate with earlier functionality and preserve existing behavior. EvoWeb evaluates this setting with cumulative, dependency-structured requests and browser tests, and finds cross-request integration especially challenging.

1 INTRODUCTION

  • Existing benchmarks cover single-request generation or refinement toward fixed intent, but not user-facing web applications evolving through successive requests.The resulting question is whether agents can extend persistent applications while integrating earlier functionality and preserving existing behavior.Source: §1 · p. 1
  • EvoWeb contains 320 cumulative requests across 32 applications, with follow-up features grounded in real-world websites and organized by functional dependencies.Agents receive requests sequentially and modify the application produced by preceding requests.Source: §1 · pp. 1–2
  • Five browser tests after each request assess new functionality, integration with an earlier dependent feature, and previously requested functionality.The dependency DAG is hidden from the agent.Source: §1 · p. 2, Figure 1 · p. 2
  • Trajectory performance is integrated across user-tolerance thresholds rather than tied to one rule for stopping after an intermediate failure.This avoids assuming either immediate abandonment after failure or unconditional continuation.Source: §1 · p. 2
  • Cross-request integration is consistently harder than implementing new or earlier-requested functionality; later requests with more or deeper dependencies are less accurate, and runtime errors rise over successive requests.Fable 5 and Kimi K3 lead with Grades of 0.694 and 0.684.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies long-horizon coding and web-generation, but EvoWeb focuses on user-facing web applications that accumulate new functionality across requests. Its dependency graph models functional prerequisites, while its scoring varies the user’s tolerance for intermediate failures.

2 RELATED WORK

  • Coding benchmarks have extended evaluation from isolated code generation to repository tasks, long-horizon software work, and persistent multi-request sequences; EvoWeb studies this problem in web development.Examples include SWE-bench, SWE-EVO, SWE-Marathon, EvoCode-Bench, SWE-Together, StaminaBench, and SlopCodeBench.Source: §2 · p. 3
  • EvoWeb differs from web-generation benchmarks centered on single requests or refinement toward fixed intent: each request adds functionality to the codebase produced by all prior requests.Its hidden DAG represents dependencies in application evolution and supports tests of cross-request integration.Source: §2 · p. 3
  • EvoWeb’s request-dependency DAG captures when later application functionality relies on earlier features, unlike DAGs that organize evaluation criteria.The graph remains hidden from the evaluated agent.Source: §2 · p. 3, Figure 2 · p. 4
  • Integrating scores over user-tolerance thresholds avoids choosing a single stopping rule, spanning fail-stop and independent scoring as endpoint policies.A threshold represents the minimum request quality a user accepts before abandoning the application.Source: §2 · pp. 3–4

3 EVOWEB BENCHMARK

EvoWeb is designed to evaluate realistic, cumulative web-app development through agentic, multi-turn work. It builds grounded request sequences with functional dependencies, tests incremental behavior in the browser, and scores trajectories across user-tolerance thresholds.

3 EVOWEB BENCHMARK

  • EVOWEB is designed to be realistic, cumulative, and agentic: requests reflect real product features, build on an evolving application, and are handled through a tool-using coding-agent scaffold.The setting tests continued correctness and coherence as the codebase grows.Source: §3 · p. 4

3.1 DATA CURATION

  • The benchmark retains 32 application trajectories with 320 requests, grounded in features from similar deployed websites and spanning 16 domains.The trajectories are selected from 40 subsampled initial specifications, then undergo automated verification and repair followed by human curation.Source: §3.1 · pp. 4–5
  • Each sequence has a hidden dependency DAG with a chain of depth at least three and at least three later requests depending on earlier functionality.Five tests assess new functionality, integration along a dependency, and an earlier request; one backward test samples earlier functionality rather than exhaustively auditing regressions.Source: §3.1 · p. 5

3.2 AGENTIC HARNESS

  • The agent iteratively inspects and modifies a persistent repository, executes commands, and revises its implementation across requests without receiving test outcomes between requests.The first request matches the original WebGen-Bench instruction, enabling comparison of the interaction protocol before cross-request dependencies arise.Source: §3.2 · p. 5

3.3 REQUEST EVALUATION

  • A browser agent executes UI tests and assigns YES, PARTIAL, or NO according to whether expected outcomes are observed.The evaluator uses a WebVoyager agent with Selenium; WebGen-Bench reported 86–94% agreement with human annotators, and EvoWeb reports a blinded human audit in its appendix.Source: §3.3 · p. 6

3.4 METRICS

  • Grade integrates delivered functionality across all user-tolerance thresholds, while request survival counts consecutive requests meeting each threshold.Request quality is the mean of its test scores; unreached requests contribute zero to delivered functionality, and uniform threshold integration weights all thresholds equally.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare 14 coding models on persistent, multi-request web applications, finding substantial performance differences and persistent gaps across user-tolerance thresholds. Results also vary by request language, while stronger scores do not simply track greater generation effort.

4 EXPERIMENTS

  • The evaluation tests 14 models on 320 requests, preserving each repository across ten requests and using a fixed interaction budget and browser evaluator.Each request allows up to 20 agent turns.Source: §4 · p. 6

4.1 MAIN RESULTS

  • Fable 5 (0.694 Grade) and Kimi K3 (0.684 Grade) lead; the weakest model scores 0.162 Grade.The top two are not ranked because their 0.010 gap is within run-to-run variation; the best survives about 7.3 of 10 requests.Source: §4.1 · p. 6, Table 2 · p. 7
  • Fable 5 leads across every user-tolerance threshold, while the gap between stronger and weaker models persists throughout the range.The exact ordering can change across thresholds.Source: §4.1 · p. 6, Figure 3 · p. 7
  • Fable 5 and GPT 5.6 Sol each produce about 100K output tokens per trajectory; Fable 5 uses 4.7 turns per request.Half of each model’s output tokens arrive in the first three turns, and Fable 5 costs the most per sample.Source: §4.1 · pp. 6–7, Figure 3 · p. 7
  • Alternative judges flip only 2–3 of 28 model-pair rankings, all among GPT 5.6 Sol, Opus 4.8, and Sonnet 4.6.The paper reports that no judge favors its own model family.Source: §4.1 · p. 7

4.2 EFFECT OF REQUEST LANGUAGE

  • Only Sonnet 4.6 stays within 0.03 of its English Grade in every language; other models vary, including drops in high-resource languages.GPT 5.6 Luna scores 0.198 in French versus 0.277 in English.Source: Table 3 · p. 7, §4.2 · p. 8

5 ANALYSIS

The analysis finds that composing features across dependency-linked requests is harder than delivering new or earlier functionality alone. Runtime errors rise as applications evolve, and models exhibit distinct patterns in their failed tests.

5.1 DAG-STRUCTURED REQUESTS ANALYSIS

  • Average Grade falls from 0.58 for backward tests to 0.55 for functional tests and 0.51 for integration tests.Integration tests traverse dependency edges; requests 8–10 also show declining accuracy with greater dependency depth and in-degree.Source: §5.1 · p. 8

5.2 BUILD AND RUNTIME ERRORS

  • Runtime errors track Grade more closely than build errors (Spearman ρ = −0.72 vs. −0.52) and become more frequent as requests accumulate.Build errors are rare for most models; Qwen3.5 35B-A3B has a 2.5% build-error rate and a 36.6% runtime-error rate.Source: §5.2 · p. 9
  • Tool-call errors do not correlate with Grade (Spearman ρ = −0.06); they cost steps rather than quality because agents can repair them on the next step.Source: §5.2 · p. 9

5.3 FAILURE MODES IN EVOLVING APPLICATIONS

  • Haiku 4.5’s failures most often involve missing data (31.2%, versus 21.3–23.6% for other models); Fable 5 has 2.2% runtime or build errors, versus 13.7–17.3%.Fable 5’s remaining failures skew toward incorrect values and styling, where it has the highest shares among the four models.Source: §5.3 · p. 9

6 CONCLUSION

EvoWeb evaluates whether coding agents can extend persistent web applications while integrating new features and preserving earlier behavior. Its experiments identify cross-request integration and sustained runtime reliability as improvement priorities; longer trajectories and closed-loop repair remain future extensions.

6 CONCLUSION

  • EvoWeb comprises 320 cumulative requests across 32 applications, grounded in real-world features and structured by functional dependencies.Browser-based tests assess intermediate applications, and tolerance-averaged metrics avoid imposing a single stopping threshold.Source: §6 · p. 9
  • Agents perform worse when composing features across requests than on new or earlier functionality alone; deeper dependencies, more dependencies, and accumulated requests bring further accuracy or runtime challenges.The authors identify feature integration and sustained runtime reliability as priorities for improving coding agents.Source: §6 · p. 9
  • Extending EvoWeb to longer trajectories and user-reported failures with repair requests would let future work examine recovery during continued application evolution.Source: §6 · p. 9

ETHICS STATEMENT

EvoWeb uses public specifications and website features, fictional seeded data, and locally run applications; it does not involve human subjects.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The appendix documents how EVOWEB’s application trajectories are selected, curated, grounded in real products, and made available with their construction and evaluation materials.

REPRODUCIBILITY STATEMENT

  • The dataset, requests, verification tests, dependency graphs, pinned seeds, and scoring code are available; the appendix also identifies where construction, evaluation, and validation procedures are described.The authors report that candidates undergo automated verification and repair, followed by human curation.Source: REPRODUCIBILITY STATEMENT · p. 10, B APPLICATION SELECTION AND DOMAIN… · p. 14

A EXAMPLE REQUEST COMPARED WITH WEB-BENCH

  • EVOWEB requests target user-facing product features, unlike the compared Web-Bench task, which is framed as a developer-facing specification.Table 6 contrasts a Web-Bench task with an EVOWEB request across audience, feature source, dependency definition, and checking method.Source: A EXAMPLE REQUEST COMPARED WITH… · p. 14, Table 6 · p. 14

B APPLICATION SELECTION AND DOMAIN COVERAGE

  • The 32 retained trajectories cover 16 of 20 WebGen-Bench domain classes, with broad application categories represented at proportions close to the source collection.User-interaction, content-presentation, and data-management applications account for 50%, 25%, and 25% of EVOWEB, versus approximately 49%, 28%, and 24% of the source collection.Source: B APPLICATION SELECTION AND DOMAIN… · p. 14, Figure 5 · p. 15

C GROUNDING: COLLECTING SIMILAR WEBSITES AND FEATURES

  • For each seed, a retrieval agent finds two similar deployed websites and records two to four plausible, implementable feature ideas from each for the request generator.The generator receives those records as its only source of feature ideas and does not see the retrieved pages.Source: C GROUNDING: COLLECTING SIMILAR… · p. 15

C.1 GROUNDING ABLATION: REALISM AND DIVERSITY

Grounding requests in real websites improves their judged realism and reduces repeated phrasing across apps; the appendix also details the generation pipeline and its feature archetypes.

C.1 GROUNDING ABLATION: REALISM AND DIVERSITY

  • The grounding ablation held the 32 seed prompts and generator fixed, removing only real-world features from the ungrounded counterpart.The preference judges were instructed to assess mature-site feature fit, not writing style, request length, or detail level.Source: §C.1 · pp. 16–17
  • 69–74%: all three judges preferred grounded requests, with 95% intervals above 50%; grounding also reduced 4-gram overlap between unrelated apps by 29%.The blind pairwise test compared grounded and ungrounded sets of follow-up requests; lower overlap indicates more diversity.Source: Table 7 · p. 16, §C.1 · pp. 16–17

D DATASET GENERATION AND VERIFICATION

  • The generator assigns follow-up requests to four archetypes, including derived-state features, interactive tools, multi-page workflows, and design restyles.Across 288 follow-up requests, the distribution was 112 derived state, 77 multi-page workflow, 67 interactive tool, and 32 design restyle.Source: D DATASET GENERATION AND VERIFICATION · pp. 17–18

D.1 GENERATOR SPECIFICATION

The appendix specifies request and test construction, evaluator validation, execution settings, variability, and tolerance-sweep reporting. It documents joint test satisfiability alongside limits on interpreting model rankings.

D.1 GENERATOR SPECIFICATION

  • Each follow-up receives three functional tests, an integration test that checks a dependency, and a backward test that rechecks earlier functionality.The generator also constructs a dependency DAG, distinct non-trivial features, pinned data, and app-wide rules; later requests rely only on user-visible behavior explicitly requested earlier.Source: §D.1 · pp. 18–19
  • 284 of 288 follow-up requests had a single application pass all five tests at the same point in a trajectory.Every test received a full score in at least two runs of the five strongest models; the remaining four requests’ tests were each passed individually, though not together in one run.Source: §D.1 · p. 19

D.2 VERIFIER CHECKS

  • The verifier audits generated sequences against 12 rules and admits a sequence only after it passes, allowing at most one repair pass.The checks include dependencies, feasibility, test structure, and implementation-agnostic expected results.Source: §D.2 · p. 20

D.3 HUMAN VALIDATION OF THE EVALUATOR

  • 92.0%: overall linear-weighted agreement between human and benchmark evaluator scores in the balanced 300-execution validation sample.Exact agreement was 95.0% for screenshot-assessed executions and 38.5% for the 39 ambiguity-selected executions requiring live inspection; navigation errors appeared only in the latter group.Source: Table 10 · p. 20, §D.3 · pp. 20–21

E EXECUTION DETAILS

  • The repository and conversation history persist across ten requests, with a budget of at most 20 agent turns per request and fresh browser sessions for tests.The evaluation uses OpenHands with its recommended configuration.Source: E EXECUTION DETAILS · p. 21

F RUN-TO-RUN VARIABILITY

  • Three-run standard deviations are coarse estimates, and open-source models were run once, so Grade gaps around 0.01 do not establish their ranking.The reported standard deviations capture run-to-run variation, not variation from choosing the 32 applications; language and DAG analyses also lack confidence intervals.Source: F RUN-TO-RUN VARIABILITY · p. 21, Table 12 · p. 21

G TOLERANCE SWEEPS FOR THE OPEN-SOURCE MODELS

  • Grade and Survival integrate delivered functionality and request survival over all tolerance thresholds; the open-source sweep figure plots both measures against tolerance.The area under each curve is the corresponding Grade or Survival reported in the table.Source: G TOLERANCE SWEEPS FOR THE OPEN-SOURCE… · p. 22, Figure 6 · p. 22

H SENSITIVITY TO USER TOLERANCE

Model rankings are mostly similar between mean request accuracy and Grade, but the fail-stop threshold can reorder models; Grade also depends on how user-tolerance thresholds are weighted.

H SENSITIVITY TO USER TOLERANCE

I ROBUSTNESS TO THE JUDGE MODEL

Alternative judges largely preserve the reported Grade ordering, while evaluation is computationally intensive but can be parallelized across isolated browser flows.

I ROBUSTNESS TO THE JUDGE MODEL

J COMPUTE COST

  • Each model is evaluated on 320 requests, with up to 20 agent turns and five browser tests per request, yielding about 1,600 browser evaluations.Source: J COMPUTE COST · p. 23
  • A single model takes about 20 hours on one machine, while parallel evaluation of multiple models finishes in under two hours using isolated flows and infrastructure scaling to hundreds of vCPUs.Source: J COMPUTE COST · pp. 23–24

Related papers

Loading 2610.16402v1…