PaperlayerView exact revision on arXiv

Source-linked AI summary

Framework and Benchmark for Code-Driven Agentic Testing in Web Development

Bin Hong, Zhenchao Zhang, Jiyuan He, Kai Zhang, Zhenya Huang

arXiv:2609.00081v1cs.SE

TL;DR

Web-application testing is difficult to evaluate because existing methods rely on predefined checklists and constrained benchmark settings, leaving VLM bug discovery insufficiently tested. The paper introduces CAT, implemented through CATJudge and CATTest, and finds that mainstream VLMs perform poorly, exposing a gap between current capabilities and practical testing demands.

  • Problem

    Existing web-application evaluations rely on predefined checklists and constrained benchmark data and frameworks, leaving autonomous VLM bug discovery systematically untested.

  • Method

    CAT uses Playwright-driven browser interaction, feedback gathering, and autonomous exploration, instantiated with CATJudge and the 102-application CATTest benchmark.

  • Results

    Mainstream VLMs perform poorly on CATTest, with R-score below 43, revealing a substantial gap from practical web-development testing requirements.

  • Takeaways & Limitations

    Bug discovery in real web projects remains a strong probe of coding intelligence beyond conventional code-generation benchmark scores.

  • Takeaways & Limitations

    The ground-truth bug set may be incomplete because bug annotation is highly labor-intensive and no single effort can exhaustively identify defects.

Abstract

from arXiv · show

End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks, leaving the bug-discovery ability of vision-language models (VLMs) systematically untested. We introduce \textbf{C}ode-driven \textbf{A}gentic \textbf{T}esting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. We release our code and data at https://github.com/SleepyWithoutCoffee/CATJudge.

1 Introduction

Existing web-application evaluations do not systematically test autonomous bug discovery: they rely on predefined checklists and limited benchmark settings. CAT addresses this gap with Playwright-driven exploration, CATJudge, and CATTest, while experiments show current VLMs remain inadequate.

  • Motivation: AI-generated web applications require visual and interactive feedback because many bugs appear only when code is rendered in a browser.End-to-end GUI testing is therefore essential for quality assurance and training-data filtering.
  • Evaluation Gap: Current testing benchmarks rely on predefined checklists, limiting evaluation to anticipated GUI operations as generated pages develop unanticipated features.Expanding checklist coverage also increases testing cost.
  • Evaluation Gap: Existing web-generation evaluations are constrained by their benchmark data and frameworks, limiting interaction complexity and bug subtlety.This leaves systematic evaluation of VLM bug discovery in intensive testing scenarios underdeveloped.
  • Approach: CAT has agents write Playwright code, gather textual and visual feedback, and autonomously explore web applications to discover bugs.CATJudge unifies Browser-Use and Computer-Use tools, while CATTest contains 102 AI-generated applications with carefully annotated bugs.
  • Results: Mainstream VLMs achieve poor CATTest performance, with R-score below 43, revealing a substantial gap from practical web-development testing requirements.The evaluation uses CATJudge on the CATTest dataset.
  • Contributions: The paper contributes a workflow model, CATJudge, CATTest, and experiments evaluating mainstream VLMs on agentic web testing.The benchmark contains 102 complex web applications for assessing CAT capabilities.

2 Related Works

Related work spans code-generation benchmarks and GUI agents, but conventional web-code evaluation often uses static tests or automated interaction within constrained settings. These limitations motivate more realistic testing of deployed web applications.

  • Code Generation Benchmarks: Code-intelligence benchmarks have progressed from function-level generation toward file-level and repository-level tasks aligned with real-world needs.They generally verify generated code with pre-written static test scripts and test cases.
  • GUI Agents: GUI agents use visual or accessibility information to interact with real environments through specialized tools.They are primarily divided into Browser-Use Agents and Computer-Use Agents.
  • Workflow: Figure 1 abstracts AI-driven web development as a workflow whose testing stage is the focus of this evaluation.The figure caption identifies the testing workflow as the evaluation target.
  • GUI Agents: Web-generation benchmarks increasingly use GUI agents because rule-based validation and LLM-as-Judge scoring struggle to reflect interactive functionality after deployment.Automated GUI interaction is intended to test pages in more realistic execution settings.

3 CAT in Web Development

CAT models AI web development as a closed loop and introduces autonomous, code-driven browser testing to uncover defects beyond predefined checks. CATJudge unifies GUI interaction tools, while CATTest provides 102 collaboratively constructed applications with complex interactions and annotated bugs.

  • 3 CAT in Web Development: The AI web-development workflow is modeled as a closed loop from user requirements through AI implementation and testing to delivery.Testing consumes information handed off from development and includes developer-side, hybrid-side, and user-side inspection layers.
  • 3 CAT in Web Development: CATJudge formulates autonomous testing as a finite-horizon POMDP whose terminal action emits a PASS or NOT_PASS verdict and discovered-bug list.The reward compares the reported bug set with the ground-truth bug set, and episodes also end at the horizon.
  • 3 CAT in Web Development: CATJudge supplies a QA-oriented project document and initial screenshot, then returns textual tool feedback and delayed page-state screenshots after interactions.The document summarizes features, technical context, state variables, and the interaction surface relevant to testing.
  • 3 CAT in Web Development: CATJudge exposes browser and computer-use GUI tools, combining encapsulated atomic operations with freely authored script execution.Its unified runtime supports Playwright in a real browser and PyAutoGUI keyboard/mouse automation on a shared virtual display.
  • 3 CAT in Web Development: CATTest contains 102 AI-generated web applications spanning 9 top-level categories, with interactions and bugs developed through close human-AI collaboration.The construction process targets broad interaction coverage, complex generation tasks, and subtle defects, followed by iterative annotation and verification.

4 Experiments

Experiments show that current VLMs struggle with autonomous bug discovery, while Hybrid-Side testing and code-oriented tool use improve testing effectiveness and efficiency.

  • Main results: 42.57 is the best R-score, while 48.56 is the best P-score, so no model reliably uncovers half the GT bugs with trustworthy reports.Claude-Opus-4.7 achieves the best R-score; Gemini-3.1-Pro achieves the best P-score.
  • Main results: 98.65 Recall for Claude-Sonnet-4.6 and 93.67 for GPT-5.4 contrast with R-scores of 37.68 and 17.58, respectively.High recall does not ensure that models identify the specific defects causing project failures.
  • Capability gap: Five frontier models exceed 36 R-score, two open models reach 33.56 and 30.78, and the remaining models fall below 25.The intermediate group is separated from the rest by roughly six R-score points.
  • Capability gap: GPT-5.5 more than doubles GPT-5.4 in R-score, reaching 39.41 versus 17.58.The reported within-family difference accompanies a broader capability gap across model tiers.
  • Workflow analysis: Hybrid-Side testing delivers the best R-score and P-score, while pure code review produces a competitive R-score but collapses in P-score.Without interactive and visual feedback, code review massively over-reports speculative defects that do not manifest at runtime.
  • Tool use: Claude-Opus-4.7 concentrates tool calls on Playwright script execution, whereas GLM-5V-Turbo distributes most calls across atomic operations.Figure 5 reports proportions of calls and total calls for each tool-set configuration.

5 Conclusion

The paper introduces CAT for autonomous Playwright-based exploration of AI-generated web applications, together with CATJudge and CATTest. Mainstream VLMs perform poorly, with a substantial gap between current capabilities and practical testing requirements.

  • Conclusion: CAT lets agents write Playwright code, gather textual and visual feedback, and autonomously explore web applications to discover bugs.The paradigm targets the gap left by checklist-based and benchmark-confined evaluation.
  • Conclusion: CATJudge unifies Browser-Use and Computer-Use tools in one isolated environment, and CATTest contains 102 web projects with annotated ground-truth bugs.These contributions instantiate the code-driven testing paradigm and provide a benchmark for evaluation.
  • Conclusion: Mainstream VLMs achieve poor CATTest performance, with the highest measured score still below the level required for unattended deployment.The conclusion also reports a sizable performance cliff separating frontier models from the rest.

Limitations

The benchmark’s ground-truth bug set may be incomplete, and the evaluation focuses on functional correctness because subjective dimensions are harder to reproduce consistently.

  • Ground-truth completeness: Ground-truth bug annotation cannot exhaustively identify every defect, so additional model-reported bugs on buggy projects are not penalized in R-score.P-score complements R-score because the ground-truth set is expected to expand through future use.
  • Subjective metrics: Subjective evaluation remains difficult because models may report minor, barely user-perceivable issues as bugs.The paper gives continued background animation after gameover as an example of behavior that may be common and acceptable.
  • Subjective metrics: The evaluation concentrates on functional correctness because subjective scoring criteria produce less reproducible results across models.Subjective dimensions are left as complementary work for dedicated benchmarks.

A.1 Statistics

CATTest comprises 102 AI-generated web applications with complex interaction-focused prompts and naturally occurring bugs. Its generated projects contain substantial bug prevalence despite using a frontier coding agent without iterative regeneration or self-repair.

  • 102 AI-generated web applications span 9 top-level categories, each organized around core interactions defining its primary input modality and feedback loop.
  • Prompt difficulty is quantified by counting user stories produced through a two-stage decomposition pipeline, with larger counts indicating richer latent requirements.
  • 22.55% is the project-level pass rate for Claude Code + Claude-Opus-4.6, with 1.86 GT bugs per project on average.
  • 100% of Marketing/Landing Page projects, 90.91% of 3D/Visualization projects, and 89.47% of Game projects contain at least one GT bug.
  • CATTest obtains GT bugs directly from AI-coding artifacts rather than manually injecting defects into otherwise correct code.

A.2 Models Involved in Bug Annotation

Bug annotation combines human verification, source-code analysis, and model-based discovery to build and validate CATTest’s bug set. The benchmark targets subtle interaction-dependent defects generated by a frontier coding agent.

  • Claude-Opus-4.7, Claude-Opus-4.6, Gemini-3.1-Pro, Gemini-3-Flash, Gemini-3.1-Flash-Lite, GPT-5.5, GLM-5V-Turbo, and Qwen3.5-Plus participate in tool-competition pooling.
  • The evaluation uses a frontier coding agent because weaker generators often produce obvious defects or projects that fail before meaningful interaction.
  • A reported bug matches a GT bug when it describes the same functional problem, a directly caused symptom, or a consequence that would disappear if the GT bug were fixed.
  • Matching permits different granularity and causal depth, including reports that cover multiple GT bugs or identify an underlying root cause.
  • Project 71 illustrates CAT’s target defect: pausing during a wave-transition countdown leaves the pause overlay stuck, requiring precisely timed interaction.

B Experiment Details

The experiments evaluate image-understanding models from several major provider families as VLM testers in CATJudge.

  • The evaluated VLMs include Claude, Gemini, and GPT model families, with eligibility requiring image-understanding capability.

C Additional Experiments

Additional experiments assess CATTest’s run stability, the value of project documentation, and the cost and capability trade-offs of Hybrid-Side testing. Code-driven interaction is especially useful for applications with opaque visual surfaces such as canvas or WebGL games.

  • Five repeated Claude-Opus-4.7 evaluations indicate that CATTest produces reproducible measurements across independent runs.
  • Removing project documentation reduces Claude-Opus-4.7’s R-score from 40.64 to 27.17 and P-score from 38.79 to 23.25.
  • Hybrid-Side testing completes projects in 9.1 minutes versus 13.9 minutes for User-Side testing, using roughly 65% of calls, 45% of tokens, and 58% of monetary cost.
  • A Playwright script can express compound interactions and aggregate observations in one turn, whereas keyboard-and-mouse primitives require many model-call turns.
  • Canvas and WebGL games can expose an opaque surface without individually addressable controls, making atomic interaction-based testing difficult.
  • GPT-5.5 explores a 2D shoot-em-up by downloading source code and treating the page as a programmable artifact rather than relying only on play.

D.2 Error Patterns

The paper identifies three recurring VLM testing failures: mirage reasoning, insufficient interaction coverage, and spec hallucination. These failures cause agents to overlook visible defects, avoid bug-revealing gestures, or report violations of undocumented expectations.

  • Mirage Reasoning: Mirage reasoning makes agents explain away plainly visible defects instead of trusting contradictory visual and textual evidence.On a flipping-card timer, the agent attributed malformed doubled digits to an animation frame and returned PASS with zero bugs.
  • Insufficient Interaction Coverage: Insufficient interaction coverage prevents agents from attempting elaborate gestures and alternative paths needed to expose gesture-shaped bugs.In the node editor example, the agent avoided drags outside the canvas, snap-enabled drags, and plain-click selection, finding only one real bug among five reported anomalies.
  • Spec Hallucination: Spec hallucination produces false positives when agents treat prior beliefs about application behavior as binding requirements despite documentation and observations.On a clean gardening e-commerce project, invented rules about cart clearing, stock decrement, and product-name truncation produced three false-positive bugs and a NOT_PASS verdict.
Loading 2609.00081v1…