PaperlayerView exact revision on arXiv

Source-linked AI summary

The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents

Achint Mehta

arXiv:2608.28795v1cs.SEcs.AI

TL;DR

The paper asks whether expanding an AI coding agent’s verification tools improves the software it ships. Across controlled builds, it finds that tools help when their reach matches how the application fails: boot checks address launch failures, while screenshots help with visible problems but not failures that can only be measured.

  • Problem

    The central question is whether expanding an agent’s tools for observing and checking its work improves the software it ships.

    Source: A. Goal · p. 1

  • Method

    The study compares controlled verification configurations, separating passive screenshot access from execution and aggregating within-model, within-task differences across runs.

    Source: D. Why a passive screenshot rather… · p. 5, §IV · p. 12

  • Results

    Across 1,116 builds, tool value depended on reach: a boot check removed nearly all launch failures, shell execution improved behavioral depth, and screenshots helped with visible but not measurable-only problems.

    Source: §VIII · pp. 24–25

  • Takeaways & Limitations

    Verification tools improve artifacts only when their reach covers the way the application fails.

    Source: §VIII · p. 25

    The study cannot determine screenshots’ real effect or whether their benefit depends on models actually inspecting what they built.

    Source: §VIII · p. 25

Abstract

from arXiv · show

Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screenshot tool. We call this set the agent's verification surface. This study asks whether increasing only that surface, with everything else held fixed, produces a matching growth in the quality of the software the agent ships. We built a minimal coding agent whose tool list is the single controlled variable and used it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric, and automatic probes stress-tested the API-observable behaviors. Verification's cheapest benefit arrives first, which is to make sure that the application comes up. Without any tools, about one build in seven fails to launch at all and a single boot probe removes nearly all of these failures at roughly 35 percent of a full shell's token cost, while the full shell multiplies the no-tools cost by 2.35. Screenshots help most where mistakes are visible (e.g. element placement, interaction), though even there the gain over a shell is modest and does not survive correction for multiple statistical comparisons. In cases where failures can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing. A verification tool improves the output artifact only where its reach covers the way the application actually fails.

I. INTRODUCTION … C. Prior Work

This study isolates an AI coding agent’s verification tools to measure how they affect shipped software quality. Its findings suggest that a tool helps when it can observe the way the application fails.

A. Goal

  • Across six models, seven applications, and 1,116 independent runs, every application received condition-blind human grading and API-observable behaviors were automatically probed.The study measured shipped output files rather than the agent’s work transcript.Source: A. Goal · p. 1

1) A controlled dose-response measurement

  • The study changes only the verification-tool list in a controlled agent, comparing an ordered ladder from no tools through shell plus screenshots.The five core configurations are no tools, linter, boot probe, full shell, and shell plus screenshots; three additional targeted configurations are described in Section II.Source: A. Goal · p. 1, 1) A controlled dose-response… · p. 2

2) The boot probe result

  • The boot probe recovered nearly all launch failures and delivered the highest artifact-quality-to-token ratio among the tested configurations.It installs dependencies, starts the server, and reports whether it listens.Source: 2) The boot probe result · p. 2

3) A more nuanced look into the question “does sight help”

  • Sight’s likely gains over a shell involve element placement and interaction, but the advantage is modest and does not survive multiple-comparison correction.Screenshots add little for measurable but invisible failures such as flickering drag-and-drop or unsmooth scrolling; static layout mistakes are often inferable from code.Source: 3) A more nuanced look into the… · p. 2

C. Prior Work

  • Prior work found that agent scaffolding affects output quality [1], while studies such as SWE-agent [2] and Agentless [3] compare complete systems using pass/fail task outcomes.This study instead varies one part of the scaffold and measures quality rather than a simple pass/fail rate.Source: C. Prior Work · p. 2
  • A previous observational study [11] found reasoning effort improved first-try reliability more than tool access, motivating this study to hold model capability fixed and measure tool-specific effects.That study reported that browser testing raised cost by 42 to 68 percent without improving functional score; the present study uses identical inputs across tool conditions.Source: C. Prior Work · p. 2
  • Prior visual-feedback studies [12], [13], [14], [15], [16], [17] focus on visually defined tasks, whereas this study also tests API-observable applications and finds visual advantage shrinks toward zero there.Separating sight from execution and measuring granular outcomes distinguishes this study from systems that provide both abilities and report task success.Source: C. Prior Work · pp. 2–3
  • Coding with Eyes [13] argues GUI code needs interaction to verify; a perfectly rendered page in this study ignored every click, showing passive screenshots provide only a lower bound on sight’s value.Unlike WebArena [18] and OSWorld [19], which vary task worlds while holding agent abilities fixed, this study holds tasks fixed and varies what the agent can observe.Source: C. Prior Work · p. 3

II. STUDY DESIGN · A. The agent · B. The applications

The study isolates verification-tool availability with a minimal controlled agent and applications designed to expose specific failure modes. Its application suite spans real-time behavior, concurrency, visual correctness, regression safety, and performance.

A. The agent

  • The minimal agent kept its prompts and environment fixed, enabling tool configuration to vary while logging model inputs, outputs, tool calls, and token usage.The design also allowed the researchers to verify whether a model called a particular tool.Source: A. The agent · p. 3
  • The agent loop allowed one fixed nudge only to prompt completion, and 29 of 1,116 runs needed it before declaring completion.The nudge did not add verification ability or change the artifact.Source: A. The agent · p. 3
  • Basic file tools were available in every condition; only the verification tools added to them varied.Source: A. The agent · p. 3

B. The applications

  • The specifications were tool-independent, and each targeted behaviors and failures that a particular kind of verification could plausibly prevent.Separating the task specifications from tool schemas let models select tools based on their reasoning rather than user instructions.Source: B. The applications · p. 3
  • The message-board spec set an easy calibration floor, while the kanban-board spec tested stressed ordering, cross-window convergence, and browser-only drag-and-drop feel.API probes exercised ordering and convergence; human grading assessed drag-and-drop feel.Source: B. The applications · pp. 3–4
  • The seat-booking spec tested whether conflicting concurrent requests could both book one seat; calendar-week-view and metrics-dashboard targeted distinct visual failures.Calendar checks concerned event widths and time-proportional placement; dashboard checks covered three screen widths, dynamic dark mode, and awkward data layouts.Source: B. The applications · p. 4
  • The brownfield kanban-labels task tested regression safety when modifying unfamiliar code, while log-explorer-perf targeted measured performance and bounded virtualized-list rows.The log viewer had over 100,000 rows, a 200-row page limit, and strict latency requirements.Source: B. The applications · p. 4

C. The tool configurations · D. Why a passive screenshot rather than browser automation · E. Why the execution configuration does not trump the other configurations

The study varies the verification tools available to coding agents and separates screenshot-based seeing from execution ability. It also explains the rationale and limitations of those controls.

C. The tool configurations

  • The first five configurations progress from no verification through static checks, boot-only probing, unrestricted execution, and execution plus screenshots.They are no_verification, static, boot_check, execution, and visual; basic file tools are available in every configuration.Source: C. The tool configurations · p. 4
  • The remaining configurations isolate screenshots without arbitrary shell access, defer shell verification until completion, or let agents run their own test command.These are visual_no_shell, delayed_verification, and behavioral, respectively.Source: C. The tool configurations · p. 5

D. Why a passive screenshot rather than browser automation

  • The visual conditions expose Playwright rendering but not its click, type, drag, or resize controls, separating seeing from acting and making each look countable.Each screenshot call takes one URL, whereas browser actions can mix looking with acting or hide many actions in one script call. An earlier study found intact interaction capability raised cost by 42 to 68 percent without improving reliability.Source: D. Why a passive screenshot rather… · p. 5
  • Click-dependent faults fall outside passive screenshots: one model installed its own scriptable browser in three runs, and models noted the limit in 21 more.Those other runs involved working around the restriction or falling back on reading code.Source: D. Why a passive screenshot rather… · p. 5

E. Why the execution configuration does not trump the other configurations

  • Shell access could recreate some tools at token cost, but self-built screenshots return bytes rather than visual input, so only the harness can show agents pixels.The run manifests record vision = false outside the two sight conditions, and command logs showed no attempt to compute screenshot facts such as dimensions or pixel colors.Source: E. Why the execution configuration… · p. 5
  • Across 252 unrestricted-shell runs without granted sight, no model gave itself sight; Gemini scripted Puppeteer in three runs but never captured or read an image.The scripts drove drag-and-drop and reordering interactions on a kanban board, but no image was read into the model’s input.Source: E. Why the execution configuration… · p. 5

F. Application and tool-configuration pairing · G. Controlled test environment

The study paired a common set of tool configurations with every application and reserved targeted configurations for selected tasks. It controlled prompts, sampling settings, and run infrastructure so that tool comparisons varied as intended.

F. Application and tool-configuration pairing

  • Five core configurations ran on every application, while three targeted configurations were limited to selected applications to control build volume and token cost.The core conditions were no_verification, static, boot_check, execution, and visual; targeted conditions were visual_no_shell, delayed_verification, and behavioral.Source: F. Application and tool-configuration… · pp. 5–6
  • The study comprised 1,116 runs, with four runs per application-condition-model combination by default and a fifth run for selected combinations.Counts were 192 each for no_verification, static, boot_check, and execution; 174 for visual; 90 for visual_no_shell; 60 for delayed_verification; and 24 for behavioral. Comparisons received equal run counts, the analysis plan was frozen before grading, and excluding fifth runs left all six primary verdicts unchanged.Source: F. Application and tool-configuration… · p. 6
  • The visual_no_shell condition isolated the value of seeing versus running on visually difficult tasks, and served as a negative control on the log explorer, where screenshots cannot reveal scrolling delays.On the log explorer, visual versus execution and the screenshot-only arm were flat or slightly negative, as expected for the control.Source: F. Application and tool-configuration… · p. 6
  • Delayed verification tested “finish first, then verify it works” on kanban-board and seat-booking, whose concurrency makes runtime behavior complex enough to hide surprises.The visually difficult calendar-week-view and metrics-dashboard were excluded because an unlocked shell cannot reveal their visual failure modes.Source: F. Application and tool-configuration… · p. 6
  • Behavioral testing ran only on seat-booking, where models could write tests for race conditions and hold expiry, unlike visual tasks or the easy message-board.The condition supplied only a command to run the model’s own test suite, which provides no feedback unless the model first writes tests; other tasks were excluded because tests cannot judge rendered geometry or the ladder already covered the message-board.Source: F. Application and tool-configuration… · pp. 6–7

G. Controlled test environment

  • All runs used identical prompt templates, environment briefings, and task specifications, verified by hashes across all 1,116 runs.Only the workspace path varied; the environment briefing described the available tools and dependency installation without favoring a configuration.Source: G. Controlled test environment · p. 7
  • Temperature was fixed at 0.0 and top_p at 1.0 when accepted, while models rejecting those settings used the same provider defaults across their own conditions.Because comparisons were within each model, sampling settings did not differ between tools-on and tools-off runs; replicates addressed remaining variation across repeats.Source: G. Controlled test environment · p. 7
  • All 1,116 runs used one API gateway, recorded model identifiers and run metadata, and ended with model-declared completion without harness or resource-limit cutoffs.Runs took place between June 12 and July 11, 2026.Source: G. Controlled test environment · p. 7

H. Analysis rules · I. Tracking Tool usage · J. Grading mechanism: Primary human grading, secondary API Probes

The study counted runs by assigned tool condition, treated non-starting applications as failures, and tracked tool use through invocation logs. It graded applications primarily with a blinded human rubric and used API probes for behaviors observable through the backend.

H. Analysis rules

  • Runs were graded by assigned condition even when models ignored granted tools, so comparisons estimate the effect of granting tools, not using them.Non-use was concentrated: grok-4.3 accounted for 55 of 100 non-use runs, while no model skipped the boot probe.Source: H. Analysis rules · p. 7
  • Non-starting applications received failing scores on every rubric item and were analyzed separately as startup failures, avoiding survivor-only comparisons.Excluding non-starting builds could compare a weak condition's luckiest launches with a stronger condition's typical runs.Source: H. Analysis rules · pp. 7–8

I. Tracking Tool usage

  • Invocation logs recorded tool calls and arguments, including refused calls, and confirmed that none of the 1,116 runs invoked tools outside its granted set.Screenshot use in the visual condition ranged from zero calls to more than twenty.Source: §I · p. 8

J. Grading mechanism: Primary human grading, secondary API Probes

  • A condition-blind human scored every application against weighted rubric criteria, chosen over an automated judge to keep evaluation independent of the tested models.The author scored each rubric item as pass, partial, or fail; scorecards omitted model and condition fields.Source: J. Grading mechanism: Primary human… · p. 8
  • Automatic API probes produced a functional score based on passed probe weight, retained boot failures as zeros, and stressed backend behavior under hostile or unlucky usage.Examples tested message delivery to connected clients, data survival after restart, and exactly one winner among eight simultaneous theater-seat holds.Source: J. Grading mechanism: Primary human… · p. 8
  • API probes were omitted for the calendar-week-view, metrics-dashboard, and log explorer because their layout or measured smoothness targets were not API-observable.Those applications received only the human rubric grade.Source: J. Grading mechanism: Primary human… · p. 8
  • Two independent routes for determining survival agreed on all 1,116 runs after machine records were corrected by hand audits.The routes used machine boot records with audit corrections and human scorecards; a released check recomputes both and reports disagreements.Source: J. Grading mechanism: Primary human… · p. 8
  • 98.8 percent of regraded rubric-item comparisons agreed, and survival calls were unchanged for all 112 sampled runs.The regrade also found weighted kappa 0.973 and ICC(A,1) 0.997 for per-run scores; it tested repeatability, not correctness.Source: J. Grading mechanism: Primary human… · pp. 8–9

K. Launch accommodations and the configuration-space rule · L. Whole Application survival · III. HYPOTHESES

The sections define fair rules for launch accommodations and whole-application survival, report that every build met its applicable architectural-fidelity maximum, and specify primary and exploratory tests of how verification tools may affect outcomes.

K. Launch accommodations and the configuration-space rule

  • Launch accommodations could select only values or paths in an artifact’s declared configuration space; if none worked, render-dependent criteria failed.Launch mechanics were excluded from the frozen rubric because defects were not checked identically across all runs; the two most common classes were instead measured mechanically over every run.Source: K. Launch accommodations and the… · p. 9

L. Whole Application survival

  • Survival requires the whole application to launch through a path its files declare; a batch-wide audit classified ten backend-booted but unrenderable builds as non-survivors.The boot probe checks only the backend; the observed frontend failures involved parse errors. Renderable applications that fail rubric criteria remain survivors, while applications that never start fail every rubric item.Source: L. Whole Application survival · p. 9
  • All builds met their architectural-fidelity maximum—1.0 on four live-update tasks and 0.75 on three without that requirement—and none substituted a forbidden technology.The static scan awards points for mandated stack components, including the realtime channel.Source: L. Whole Application survival · pp. 9–10

III. HYPOTHESES

  • The six primary hypotheses were specified in a written analysis plan after grading but before inferential computations, with permutation tests and Holm correction across the family.A primary result counted as supported only if it survived the correction.Source: §III · pp. 10–11
  • P1–P3 test execution versus no tools on functional scores for three API-checkable apps, boot-check versus no tools on survival across seven apps, and execution versus no tools on token cost.The P1 functional probes test behaviors such as simultaneous seat holds, data persistence after process termination, and message delivery to connected clients.Source: §III · p. 10
  • P4–P6 test visual versus execution on interface scores for two visually hard tasks, modification versus equivalent new-build tool effects, and execution versus no tools on the 100,000-row performance task.For P5, the plan expected a near-zero difference and defined support as a confidence interval close to zero.Source: §III · p. 10
  • Exploratory S1–S3 compare execution with boot-check, visual-without-shell with execution, and visual with execution on the performance task as a negative control.These secondary contrasts decompose or probe primary results and are reported without Holm correction, so they are suggestive rather than confirmatory.Source: §III · p. 11
  • Exploratory S4–S6 examine linter effects on behavior and survival, self-authored tests versus execution, and per-model sight effects alongside screenshot use.The plan asks whether self-authored tests can substitute for a shell and whether screenshot gains concentrate in models that actually look.Source: §III · p. 11

IV. RESULTS · A. More verification, better outcomes, mostly at the bottom

Verification improved launch survival, API-probed behavior, and human-judged interface quality, with the largest gains generally arriving at the bottom of the tool ladder. The outcomes stopped improving at different rungs, and the static linter/type-checker condition showed no measurable benefit over using no tools.

IV. RESULTS

  • Results pool run-weighted differences calculated within each model-task combination; stratified permutation tests with Holm correction and a mixed-effects cross-check agreed on effect directions and sizes.Source: §IV · p. 12, Table IV · p. 12

A. More verification, better outcomes, mostly at the bottom

  • Execution vs no_verification improved API functional score by +12.3 points (95% CI [+7.0, +17.5], corrected p < .001), while boot_check vs no_verification raised survival by +13 percentage points (95% CI [+10, +16], corrected p < .001).Source: Figure 2 · p. 13, Figure 1 · p. 13, A. More verification, better outcomes,… · pp. 13–14
  • Survival was 86.5% for no_verification, 84.4% for static, and 99.5% for boot_check; 56 of 66 failures came from the two conditions unable to run anything.Audited failures included corrupted, unparseable source files and database seed scripts that crashed at first boot.Source: A. More verification, better outcomes,… · p. 13, Figure 1 · p. 13
  • Static did not measurably improve survival or behavior over no_verification, and five of the study’s ten frontend parse-error failures occurred under static despite linting being intended to catch such errors.The linter was called once in an example build but was not wired up, and the agent did not check again.Source: A. More verification, better outcomes,… · p. 13
  • Human interface scores on the two visually difficult tasks rose from 72.8 with no_verification to 89.5 with boot_check, 89.2 with execution, and 96.1 with visual, but the visual-over-execution gain did not survive multiple-comparison correction.Source: Figure 3 · p. 14, A. More verification, better outcomes,… · p. 14
  • Of the 12 functional points separating blind builds from shell builds, boot_check supplied 10, while shells added the remainder on demanding concurrency, restart-persistence, and ordering-under-load probes.On the top-weight probe types, shell conditions passed 92.0% of probe weight, boot_check 91.1%, and blind or linter-only conditions 89.1%.Source: A. More verification, better outcomes,… · p. 14
  • On calendar-week-view, boot_check scored 96.6 on human interface quality versus 86.4 for execution, with greater execution-run variability (SD 31.0 vs 3.3) and two models’ shell builds pulled down by interaction defects.Source: A. More verification, better outcomes,… · p. 14

B. The boot probe: the cheapest useful verification

The boot probe nearly eliminates launch failures at far lower token cost than the full shell, while coming within a few functional points on API-checkable tasks. Its boundary is that confirming a server starts does not establish that requests work or that the page renders.

B. The boot probe: the cheapest useful verification

  • 191 of 192 boot_check runs launched at 214 thousand median tokens per build, versus 615 thousand for execution; API-task functional scores were 91.7 versus 94.2.The boot probe sits on the cost-versus-machine-probed-score efficient frontier and costs roughly a third as much as the shell.Source: B. The boot probe: the cheapest useful… · pp. 14–15, Figure 4 · p. 15
  • Boot_check trailed execution on concurrency, dashboard layout, and log-explorer smoothness; dashboard human-interface scores were 82.5 versus 92.0, and measurement-task scores were 84.8 versus 93.8.The human-score gaps were around nine points on the dashboard and log explorer.Source: B. The boot probe: the cheapest useful… · p. 15
  • Boot_check confirms only that a server starts and accepts connections, not that requests work or a page renders.One build passed the probe despite a frontend parse error preventing rendering; an SQL endpoint crash and a permanently false error banner also passed.Source: B. The boot probe: the cheapest useful… · p. 15
  • The shell's API-functional advantage over boot_check was 2.4 points (95 percent interval [-0.4, +5.2]) at 2.9 times the token cost.The interval means the shell's advantage on these tasks may be zero and is at most about five points.Source: B. The boot probe: the cheapest useful… · p. 15

C. Cost and variance

Verification costs rise steeply with surface size, while the boot probe is both cheaper and more predictable than building blind. Delayed verification is the most expensive and variable configuration, largely because it triggers repair cycles.

C. Cost and variance

  • 214k tokens for boot_check was the lowest median, versus 262k for no_verification; execution cost 615k, visual 674k, visual_no_shell 659k, and delayed_verification 1,199k.Execution cost 2.35 times the no_verification baseline; visual cost about a tenth more than execution, while visual_no_shell cost about the same as visual.Source: C. Cost and variance · p. 15
  • Boot-probe runs cost less mainly by taking fewer steps and rereading less conversation: median 17 versus 19 steps and 195 thousand versus 247 thousand input tokens.The saving is descriptive, not tested, and persists among runs that started: boot_check cost about 49 thousand tokens less than working blind.Source: C. Cost and variance · p. 15
  • 45 of 60 delayed_verification runs changed their code before declaring completion again, while the 15 unchanged builds scored about seven functional points higher.Most extra cost in this configuration went to a repair cycle; unchanged completion was described as a mild sign of a better build.Source: C. Cost and variance · pp. 15–16
  • 47K tokens was boot_check’s median cost spread, compared with 408K for delayed_verification; boot_check was most predictable and delayed_verification least predictable.Spread was computed as the standard deviation across four or five identical repeat runs for each model-application-condition combination, then summarized by condition median.Source: C. Cost and variance · p. 16, Table VI · p. 16
  • The cost-quality frontier retained its shape across behavioral probes and human interface grades, and the same shape recurred when plotted separately for each model.The study kept the two quality judgments separate because they measure different things, combining them would require arbitrary weighting, and separate views allow comparison of frontier shape.Source: C. Cost and variance · p. 16

D. The Sight Boundary

Screenshots generally improved visually graded work over shell execution, especially on tasks with visible mistakes, but the pooled advantage was suggestive rather than statistically established. Gains did not track how much models used screenshots.

D. The Sight Boundary

  • P4 pooled visual-versus-execution comparisons at +6.9 points (95% CI [+0.8, +13.4], p = .041), but the six-way Holm-corrected p = .083; visual_no_shell versus execution was +5.0 [−1.4, +11.8].Source: D. The Sight Boundary · p. 16, Figure 7 · p. 18
  • On calendar-week-view, visual and visual_no_shell beat execution by +10.9 and +4.3; on metrics-dashboard, their gains were +2.9 and +5.7.Source: D. The Sight Boundary · p. 16, Table VII · p. 17
  • For a model that took no screenshots, the pre-specified per-model sight effect was untestable; its displayed value reflects its unusually low shell-only score, not seeing the page.Source: D. The Sight Boundary · p. 18
  • Sight's descriptive gains tracked mistake visibility: +10.9 on calendar-week-view, +2.9 on metrics-dashboard, and −2.2 on the log explorer; dashboard visual_no_shell scored 97.7 versus 94.9 for visual.Calendar-week-view mistakes included misplaced events and broken interactions; log-explorer lag and stutter cannot be seen in a still image.Source: D. The Sight Boundary · p. 18
  • Grok-4.3 gained 42.9 calendar-week-view points after one screenshot and 35.6 dashboard points with none; claude-4.6-sonnet gained 23 calendar points after 62 screenshots, and no other per-model contrast was significant.The model-level pattern did not show gains increasing with screenshot use.Source: D. The Sight Boundary · p. 18, Figure 8 · p. 19
  • Gains aligned with low shell baselines rather than screenshot counts, and grok-4.3's improvement could reflect unusually bad shell builds or merely receiving the screenshot tool; the study cannot distinguish these explanations.The authors conclude they cannot credit the pooled benefit to models actually looking at their pages and fixing what they saw.Source: D. The Sight Boundary · pp. 16, 19

E. Brownfield modification: a clean null · F. Performance budgets: measurement beats sight

Verification mattered about the same for modifying unfamiliar code as for writing fresh code, while screenshots added nothing on a performance task whose failures were invisible in still images. The results support a qualified rule: checks help when they can observe the failures a task produces.

E. Brownfield modification: a clean null

  • Modification and greenfield tool gaps were similar: regression scores differed by 10.0 versus 10.6 points, and feature scores by 14.0 versus 11.8 points.The first value in each pair is the modification task’s execution-versus-no_verification gap; the second is the greenfield gap for the same board.Source: E. Brownfield modification: a clean null · p. 19, Figure 9 · p. 19
  • The functional-score difference-in-differences was −0.7 points (95% CI [−13.7, +12.7]), consistent with no difference but too imprecise to establish equality.The interval rules out a large difference of roughly thirteen points or more in either direction, while smaller differences could remain hidden.Source: E. Brownfield modification: a clean null · p. 19
  • The brownfield null is limited to a friendly starter codebase of a few files that was well structured and freshly seeded; larger or messier codebases may differ.Source: E. Brownfield modification: a clean null · p. 19
  • A mechanical check found that 140 of 144 modification builds kept and extended the starter, touching a median of three of its six files in every condition.Four builds were classified as rewrites, distributed across static and visual conditions rather than concentrated where tools might make rewriting easier.Source: E. Brownfield modification: a clean null · pp. 19–20

F. Performance budgets: measurement beats sight

  • Log-explorer-perf had 13 of the study’s 66 genuine launch failures, with 11 of 13 in no_verification and static; execution’s effect versus no_verification was +15.3 points.The effect’s interval was [+7.3, +24.0], raw p = .023, but it did not survive family correction (corrected p = .068). The task required loading a database and running query-serving code before the page could appear.Source: Figure 10 · p. 20, F. Performance budgets: measurement… · p. 20
  • On log-explorer-perf, adding screenshots to the shell scored −2.2 against the shell (95% interval [−12.5, +7.3]); visual_no_shell averaged 91.2 versus execution’s 93.8.The committed contrast and the screenshot-only comparison both show no benefit on this task.Source: F. Performance budgets: measurement… · p. 20
  • Log-explorer-perf failures required running the app with substantial data to expose frozen scrolling or an unvirtualized list, so screenshots could not reveal them.The task was designed as a negative control: its failures do not show in screenshots or register through the API, and the control behaved as predicted.Source: F. Performance budgets: measurement… · p. 20
  • Gemini-3.1-pro accounted for almost all of that apparent edge: all its no_verification builds failed to start and scored zero, while execution builds ran, producing a 100-point gap.Because exact speed targets lacked a measuring tool, the human grader assessed visible performance signs rather than timed measurements.Source: F. Performance budgets: measurement… · p. 20

G. Self-authored tests are not a substitute

Models given only their own test suites did not reach the quality of models given an execution shell. Their weaker results reflected both fewer applications launching and tests that could reinforce the model’s mistaken understanding of the task.

G. Self-authored tests are not a substitute

  • The shortfall reflected both fewer builds launching, which tests cannot fix, and self-authored tests passing when they share the model’s mistaken understanding of the task.Surviving builds also struggled on deep concurrency probes.Source: G. Self-authored tests are not a… · p. 21
  • On seat-booking (24 runs), self-authored tests yielded 83% survival and a 71.2 functional score, versus 87.3 for execution on the same task, and were the study’s weakest configuration.Execution scored 94.2 when pooled across all API tasks.Source: G. Self-authored tests are not a… · p. 21, Figure 11 · p. 21
  • Behavioral runs ended voluntarily after 22 steps and 369 thousand tokens, versus 49 steps and 810 thousand for shell runs, while writing a similar amount of code.Every run declared itself finished; none was cut short by a harness or token limit.Source: G. Self-authored tests are not a… · p. 21
  • The prespecified behavioral-minus-execution contrast was −16.1 functional-score points (95% interval [−31.7, −1.7], p = .043), negative for five of six models, but is a secondary finding from one task.The interval supports the direction of the gap more clearly than its size.Source: G. Self-authored tests are not a… · p. 21

V. THE DEFECT CATALOG: HOW APPLICATIONS ACTUALLY FAIL … B. Model signatures (exploratory, observational)

The catalog shows that applications can fail when verification covers one layer but not the one where the defect appears. Launch hazards vary with tools, while most tracked engineering choices are more strongly associated with the model that built the app.

V. THE DEFECT CATALOG: HOW APPLICATIONS ACTUALLY FAIL

  • The defect examples show that checks at one layer can miss consequential failures at another, including swallowed clicks, broken development serving, and request-time crashes.The catalog is illustrative, not a census; examples also include a date conversion that prevents rendering, an ignored lint failure followed by a syntax error, and a persistent false error banner.Source: §V · pp. 21–22
  • The same Vite misconfiguration recurred in 52 runs: 20 proxy collisions, almost all claude-4.8-opus, and 32 wrong-folder launches, almost all claude-4.6-opus.Source: §V · p. 22

A. Declared-launch integrity

  • The authors identified recurring launch problems during hand-grading, then ran code-only wiring and data-directory scanners across all 1,116 builds, with unresolved cases classified by hand.The wiring scanner checked whether startup loaded the proxy configuration and whether it pointed to the backend port; the other checked whether code created the database directory before use.Source: A. Declared-launch integrity · p. 22
  • 45 of 1,116 builds (4 percent) had wiring problems, while data-directory bugs affected 169 builds (15 percent), falling to 5.7 percent under the boot probe (Fig. 12).Wiring problems were roughly 1.6 to 6.8 percent across conditions and highest in boot_check; data-directory bugs were about 21 to 26 percent in no_verification and static, and 11 to 13 percent in shell conditions.Source: Figure 12 · p. 22, A. Declared-launch integrity · p. 22

B. Model signatures (exploratory, observational)

  • A structural scan measured ten engineering choices per artifact and compared each choice’s Cramér’s V association with model versus condition.Tracked choices included project layout, module system, process orchestration, frontend build invocation and API calls, TypeScript, tests, and README presence.Source: B. Model signatures (exploratory,… · pp. 22–23
  • Seven of ten structural choices were predicted far better by model than tool condition; whether an app shipped tests instead tracked the behavioral condition’s requirement to write tests.The tests result reflects the study design: the behavioral condition’s only tool ran the project’s own test suite.Source: B. Model signatures (exploratory,… · p. 23
  • gpt-5.5 used a src tree in 77 percent of runs and bare frontend-build invocation in 94 percent, while grok-4.3 mixed root and nested manifests in 82 percent.Launch hazards also clustered by model: claude-4.6-opus accounted for 16 of 22 proxy-not-loaded cases and claude-4.8-opus for 14 of 16 api.js collisions; mean code volume stayed 716 to 800 lines across five core conditions, while shell cost 2.35x as many tokens as no tools.Source: B. Model signatures (exploratory,… · p. 23
  • Model signatures are observational patterns, not proven causes, because models were fixed groups rather than randomized treatments and their run counts were not perfectly matched.The authors argue these habits mechanically explain why defects cluster by model, since recurring project-building choices can generate recurring defects.Source: B. Model signatures (exploratory,… · p. 23

VII. LIMITATIONS … E. Statistical power

These limitations bound what the study can conclude: screenshots cannot observe interaction-only failures, rubric-only tasks may overcredit functionally empty apps, and the grading and statistical-power design constrain certainty.

A. Passive Screenshot tool

  • 214 of 264 visual-condition runs took screenshots, but 512 of 1,326 calls went beyond the app’s root URL and 67 runs created photo-only pages to reach otherwise inaccessible states.Agents also forced state through URL parameters or temporarily edited the app to make states photographable.Source: A. Passive Screenshot tool · p. 23
  • Because the screenshot tool cannot click, type, drag, hover, or resize, it cannot find click-only faults; the study’s visual conditions therefore measure the low end of what sight can do.The inert calendar that rendered correctly but ignored clicks survived the visual condition.Source: A. Passive Screenshot tool · pp. 23–24

B. Reasoning is not carried across turns

  • Reasoning was not carried across turns in any condition, but shell-enabled arms ran longer, so preserved reasoning could widen quality gaps and narrow cost gaps.The limitation bounds what the results speak to, though the missing continuity did not distort the comparisons among conditions.Source: B. Reasoning is not carried across turns · p. 24

C. Single Grader

  • 98.8% of item verdicts were reproduced on condition-blind regrade (weighted kappa 0.97 pooled and 0.94 on launched runs), demonstrating stability but not correctness.Only human quality ratings are exposed to this concern; survival calls, crash types, and automated behavior tests are code-measured and repeatable.Source: C. Single Grader · p. 24

D. Rubric-only tasks

  • Three tasks—the calendar, dashboard, and log explorer—had no automated functional tests, and the rubric could overrate a correctly rendered but functionally empty app.The researchers flagged affected runs in their notes rather than changing the scores.Source: D. Rubric-only tasks · p. 24

E. Statistical power

  • Four to five replicates per cell are not independent chances to observe defects when repeated runs of the same model and task tend to share code quirks.The study’s statistical strength comes from six different models rather than replicates within each cell.Source: E. Statistical power · p. 24
  • Comparisons pooled across six models have reasonable power, but within-model comparisons rest on only four or five runs per condition, so per-model figures are descriptive.The study treats the six models as its independent blocks.Source: E. Statistical power · p. 24

F. Correction and null results … DATA AND CODE AVAILABILITY

Across 1,116 controlled builds, verification gains fell into distinct tiers: a tool changed the artifact only when its reach covered the way the application failed. Boot checks helped with launch failures, shells with runtime behavior, and screenshots with visible defects.

F. Correction and null results

  • Two of six pre-specified effects lost significance after correction; the wide-interval brownfield null rules out only large effects, not small or moderate ones.Effects that survive correction are supported by this design; a null does not establish that an effect is zero.Source: F. Correction and null results · p. 24

G. Non-customizable sampling setting

  • Three of six models’ APIs rejected the requested sampling settings, but each affected model used the same provider defaults across all its runs.Thus, settings differed across affected models but were consistent within each model.Source: G. Non-customizable sampling setting · p. 24

VIII. CONCLUSION

  • About one in seven blind builds failed to launch, while one cheap boot probe removed nearly all such failures; one in 192 still had a broken frontend over a healthy backend.The boot check watched whether the server started, so it could not detect that broken page.Source: §VIII · p. 24
  • Runtime behavioral depth required running the code: shell-equipped builds handled demanding conditions that cheaper conditions did not, while self-written tests stopped early after passing.On the one self-testing task, that condition used less than half the shell’s tokens and stopped after 22 steps versus 49; self-tests can repeat a misunderstanding of the task.Source: §VIII · pp. 24–25
  • Screenshots helped on average when defects were visually apparent and added nothing for measurable but invisible problems, but the average benefit did not survive corrected significance testing.Most of the benefit came from one model that barely looked at the screenshots, leaving sight’s true effect and mechanism unresolved.Source: §VIII · p. 25
  • The strongest model gained nothing from tools on API tasks, while the weakest gained 37 points with a shell; tool value and the best choice varied by model.A boot probe watches startup, a shell drives the running application, screenshots reveal visible defects, and a linter inspects only wired code—nothing in a quarter of its runs. Choose tools by likely failure mode, not tool quantity.Source: §VIII · p. 25

DATA AND CODE AVAILABILITY

  • The study graded finished apps with identical prompts, pre-specified handling rules, and condition-blind human judgment; its data, builds, logs, scores, rubric, and analysis code are openly archived.The Zenodo archive is identified by DOI 10.5281/zenodo.21961590 and includes the dataset of record, freeze-20260720, plus the code and artifacts needed to reproduce the analysis.Source: §VIII · p. 25, DATA AND CODE AVAILABILITY · p. 25

VIII. CONCLUSION

  • Across 1,116 controlled builds, verification value separated into tiers because tools changed artifacts only when their reach covered the way applications failed.Boot probes reached startup failures, shells reached runtime behavior, and screenshots reached visible defects.Source: §VIII · p. 25

APPENDIX A · APPENDIX B · RELEASED ARTIFACT MAP

The appendices document the fixed agent interface, identify archived per-condition materials, and map the released dataset and reproduction instructions.

APPENDIX A

  • The base prompt and universal environment briefing were byte-exact across runs, with hashes verified against every run manifest.The base prompt was constant across every run, and the environment briefing was identical in every condition.Source: Appendix A · p. 25
  • The universal briefing stated that only the listed tools were available and directed agents to use what helped them complete and check their work.Source: Appendix A · p. 25

APPENDIX B

  • The archive’s agent-interface directory contains eight per-condition tool-schema files and one full example trace per condition.Source: Appendix A · p. 25

RELEASED ARTIFACT MAP

  • The paper’s source materials are published in an archive identified by Zenodo DOI 10.5281/zenodo. 21961590 and browsable at the listed GitHub repository.Source: RELEASED ARTIFACT MAP · p. 26
  • The dataset of record is the SHA256-sealed snapshot 20260720/.Source: RELEASED ARTIFACT MAP · p. 26
  • The archive README.md provides recipes to regenerate the analysis table byte-for-byte, recompute every statistic, and regrade any artifact from scratch.Source: RELEASED ARTIFACT MAP · p. 26

Related papers

Loading 2608.28795v1…