PaperlayerView exact revision on arXiv

Source-linked AI summary

The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents

Achint Mehta

arXiv:2608.28795v1cs.SEcs.AI

TL;DR

The paper asks whether expanding an AI coding agent's verification surface improves shipped software when everything else is held fixed. Using a controlled minimal agent across 1,116 applications, it finds that verification helps chiefly when a tool can reach the application's failure mode: boot checks cheaply prevent launch failures, shells add behavioral depth, and screenshots offer limited benefits for visible problems but none for purely measurable ones.

  • Problem

    The study addresses limited evidence about which parts of an AI coding agent's verification surface buy which kinds of software quality when model capability and inputs are held constant.

  • Method

    A custom agent varied only its verification tools across six models and 1,116 applications, using fixed inputs, human rubric grading, and automatic API probes.

  • Results

    Verification benefits separated by reach: boot probes nearly removed launch failures, shells added behavioral depth, and screenshots helped modestly on visible tasks but not on measurable-only failures.

  • Takeaways & Limitations

    A verification tool improves the shipped artifact only when its reach covers the way the application fails.

  • Takeaways & Limitations

    The screenshot benefit did not survive correction for multiple comparisons, and the data cannot settle the size or mechanism of sight's effect.

Abstract

from arXiv · show

Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screenshot tool. We call this set the agent's verification surface. This study asks whether increasing only that surface, with everything else held fixed, produces a matching growth in the quality of the software the agent ships. We built a minimal coding agent whose tool list is the single controlled variable and used it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric, and automatic probes stress-tested the API-observable behaviors. Verification's cheapest benefit arrives first, which is to make sure that the application comes up. Without any tools, about one build in seven fails to launch at all and a single boot probe removes nearly all of these failures at roughly 35 percent of a full shell's token cost, while the full shell multiplies the no-tools cost by 2.35. Screenshots help most where mistakes are visible (e.g. element placement, interaction), though even there the gain over a shell is modest and does not survive correction for multiple statistical comparisons. In cases where failures can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing. A verification tool improves the output artifact only where its reach covers the way the application actually fails.

I. INTRODUCTION

The study isolates verification surface as a causal variable by holding the coding agent and task inputs fixed while varying its checking tools. Across 1,116 applications, tool benefits depended on whether a tool could reach the application's failure mode.

  • The study asks how an AI coding agent's verification surface affects shipped software quality rather than its working transcript.
  • The custom agent varied only verification tools across five core configurations, from no tools through a shell plus screenshots.The core ladder was none, linter only, boot probe only, full shell, and shell plus screenshots.
  • 1,116 applications were built across six models, then evaluated by condition-blind human grading and automatic API probes.
  • A boot probe recovered nearly all launch failures and delivered the highest artifact-quality-to-token ratio.It checks dependency installation, server startup, and whether the server listens.
  • Screenshots offered modest, task-dependent gains over a shell for visible placement or interaction errors, but not for failures measurable only through behavior.The visual advantage did not survive correction for multiple comparisons, and screenshots added minimal value for invisible issues such as nonsmooth scrolling.
  • The paper extends prior whole-system comparisons by holding model capability and inputs constant while measuring which verification tools buy which quality.

II. STUDY DESIGN

The study uses a deliberately minimal agent so that prompts, environment instructions, control flow, and basic file access remain fixed while verification tools vary. This design prevents hidden agent behavior from becoming an alternative explanation for tool effects.

  • The agent uses one fixed base prompt, one fixed environment briefing, and task text, with nothing else reaching the model.
  • The model repeatedly calls available tools until declaring completion through a special finish tool, with at most one fixed reminder.
  • Basic file tools were available in every condition; only the additional verification tools differed.
  • The agent requested no extended-reasoning mode, and no reasoning content was exposed to or carried forward by the agent.Models could still reason internally, and those tokens were included in reported totals.

B. The applications

The applications were designed around distinct failure modes so that each verification tool could plausibly reach some failures but not others. The study therefore spans API behavior, concurrency, visual interaction, performance, and tool-use boundaries.

  • Each specification targeted behaviors and failures that a particular verification method could plausibly prevent.
  • The kanban board combines stressed fractional ordering, real-time convergence, and drag-and-drop feel, separating API-testable behavior from browser-only interaction.
  • The log explorer targets failures observable only through measurement, including smooth scrolling and bounded rendering over more than 100,000 rows.
  • The tool ladder ranged from building blind to screenshots, delayed shell access, and self-authored tests.The targeted conditions ask whether visual inspection, a final debugging pass, or self-testing can substitute for granted verification.
  • The screenshot channel was intentionally passive: it returned a rendered image but did not expose browser interaction.This isolated seeing from acting and made screenshot calls countable.
  • Passive screenshots cannot reach faults that appear only after clicking; one model installed its own browser in three runs, and models noted the restriction in 21 further runs.
  • Among 252 unrestricted-shell runs without granted sight, only three scripted a browser, and none captured or read back screenshots.

F. Application and tool-configuration pairing

Tool configurations were paired with applications according to the failure modes they could test, while targeted conditions reduced cost and enabled controls. The design also preserved condition assignment and consistent evaluation despite unequal tool use.

  • The five core conditions ran on every application, while visual_no_shell, delayed_verification, and behavioral ran only on targeted subsets to control cost.
  • The study contained 1,116 runs, with 192 each in no_verification, static, boot_check, and execution, plus targeted counts for the other conditions.
  • visual_no_shell was assigned to visually hard tasks to separate the value of seeing from the value of running code.It was also used on the log explorer as a negative control because still images cannot show scrolling delay or stutter.
  • The log-explorer control produced flat or slightly negative screenshot contrasts, as expected when the relevant failures are invisible in still images.
  • Delayed verification targeted kanban-board and seat-booking because their concurrency behavior could hide runtime surprises, while behavioral testing was limited to seat-booking.
  • The environment used identical prompt, briefing, and task text across conditions, verified by hashes across all 1,116 runs.
  • Applications were graded by assigned condition even when models ignored granted tools, avoiding survivor bias from scoring only runs that used verification.

I. Tracking Tool usage

The study logged tool invocations and used blinded human grading, automatic probes, and regrading to measure artifact quality while auditing launch and configuration issues.

  • Tool tracking: Tool-invocation logs recorded arguments and confirmed across all 1,116 runs that agents used only tools granted by their condition.Per-run counts included linter, boot-probe, and screenshot calls.
  • Human grading: A single grader scored every application against a frozen rubric using pass, partial, or fail judgments.Scores were produced under shuffled, opaque identifiers with model and condition fields removed.
  • Automatic checks: Automatic probes independently tested API-observable behavior, while applications that failed to boot retained zero scores rather than being excluded.Three layout- or smoothness-focused applications had no API probes and relied only on human grading.
  • Cross-checks: Human and machine assessments cross-checked boot survival, including manual review of every machine-recorded boot failure.Human grading could confirm failures or identify declared launch paths that successfully started the application.
  • Reliability: A blind regrade of 112 runs assessed the stability of the single-grader instrument, with 18 reopened records excluded from the sampling frame.The regrade used fresh opaque identifiers, shuffled order, and the same rubric and accommodation rules.
  • Launch handling: Launch-script inconsistencies were treated as model-authored defects, but the frozen rubric did not add a separate launch-mechanics penalty.Accommodations could use only configuration values or paths declared by the artifact itself.

L. Whole Application survival

Whole-application survival required a declared path to render the application, so the study combined backend boot checks with human detection of frontend failures and standardized comparisons across model-task cells.

  • Definition: Survival required the application as a whole to come up through at least one path declared by its own files.Backend boot alone could miss frontend parse errors that prevent any declared path from rendering the page.
  • Integrity checks: Architectural-fidelity checks reached the mandated stack across all builds, with live-update applications scoring 1.0 and the remaining applications reaching their 0.75 maximum.No build substituted forbidden database or transport technologies.
  • Primary hypothesis: The boot-probe hypothesis tested whether a single start/no-start signal could increase survival relative to no verification.The probe reports whether the server listens and repairs nothing itself.
  • Multiplicity: Secondary contrasts were exploratory and uncorrected, whereas the six primary contrasts received Holm correction across the hypothesis family.The paper treats secondary findings as suggestive rather than confirmatory.
  • Aggregation: Reported condition differences were computed within each model and task, then combined as run-weighted averages across all model-task combinations.This normalization prevents strong or weak model baselines from distorting tool comparisons.
  • Statistical analysis: Stratified permutation tests shuffled condition labels only within matching model-task groups, using 10,000 permutations and bootstrap confidence intervals.A mixed-effects regression independently cross-checked the six primary contrasts and matched permutation estimates to two decimal places.

A. More verification, better outcomes, mostly at the bottom

Verification improves survival, API-tested behavior, and human-graded interface quality, but the largest gains arrive when agents first receive a way to execute their applications. Further tools help only when they can reach the remaining defects.

  • Every outcome improves as verification increases, with the largest jump from no execution to a single boot probe.The study measures survival, machine-probed behavior, and human-judged interface quality across the five core conditions.
  • Tool specificity: The linter makes no measurable difference, and linter-equipped builds still contain frontend parse errors that prevent rendering.Five of the study’s ten non-rendering frontend builds occurred in the static condition, despite parse-error detection being a linter’s intended function.
  • Survival: Boot checking raises survival from 86 to 99.5 percent, after which stronger tools add no further launch benefit.The boot probe removes nearly all remaining launch failures, while execution and screenshots do not improve survival further.
  • API-probed behavior: The boot probe supplies ten of the twelve functional points separating blind builds from shell builds; the shell adds depth mainly on concurrency, restart, and ordering probes.These demanding probes involve simultaneous requests, unexpected termination with restart, and ordering under load.
  • Human-graded interface: On visually difficult tasks, scores rise from 72.8 with no tools to 96.1 with screenshots, but the screenshot gain over a shell does not survive multiple-comparison correction.The intermediate scores are 89.5 with a boot probe and 89.2 with a shell.
  • Overall pattern: Across the three measures, each rung stops helping at a different point because added tools improve only defects within their verification reach.Survival plateaus after boot checking, machine-probed behavior gains little from the shell, and human interface quality is the only measure still rising above it.

B. The boot probe: the cheapest useful verification

The boot probe delivers most of verification’s early value by nearly eliminating launch failures at low cost, but it cannot verify behavior after startup or visual correctness.

  • B. The boot probe: the cheapest useful verification: The boot probe nearly eliminated launch failures and came within a few functional points of the full shell at roughly a third of its token cost.The probe’s benefit is concentrated in ensuring that the application starts, not in checking everything it does afterward.
  • B. The boot probe: the cheapest useful verification: The cost-quality frontier places the boot probe cheaper than building blind and within a few points of the shell on API tasks.The frontier uses median token cost and machine-probed behavioral score, pooling the six models.
  • B. The boot probe: the cheapest useful verification: The probe confirms that the server listens but cannot determine whether requests are correct or whether the page renders.A frontend parse error, an SQL type error, and a permanently false error banner all passed through boot verification.
  • B. The boot probe: the cheapest useful verification: 191 of 192 boot_check runs came up, costing 214 thousand median tokens versus 615 thousand for execution, while functional scores were 91.7 versus 94.2.These scores cover the three API-checkable tasks: message-board, kanban-board, and seat-booking.
  • B. The boot probe: the cheapest useful verification: On visually hard tasks, boot_check trailed execution by 9.5 points on the dashboard and 9 points on the measurement task.The reported dashboard scores were 82.5 for boot_check and 92.0 for execution; the measurement-task scores were 84.8 and 93.8.

C. Cost and variance

Verification cost rises as the surface expands, while the boot probe is both unusually cheap and predictable; larger configurations add cost and variability without a uniformly proportional quality gain.

  • C. Cost and variance: Boot_check cost 214k median tokens, making it the only configuration cheaper than no_verification at 262k; execution cost 615k, or 2.35 times the baseline.Visual cost 674k was about a tenth above execution, while visual_no_shell cost 659k, showing that sight without a shell was not cheaper.
  • C. Cost and variance: The delayed_verification condition cost 1,199k median tokens, with 45 of 60 runs changing code after shell access unlocked.The 15 unchanged builds scored about seven functional points higher than the 45 edited builds.
  • C. Cost and variance: Boot-probe runs used 17 median steps and 195 thousand input tokens versus 19 steps and 247 thousand input tokens for working blind runs.The saving was attributed descriptively to less conversation rereading, not to writing substantially less code.
  • C. Cost and variance: Boot_check had the smallest median cost spread at 47K tokens, whereas delayed_verification had the widest at 408K.Execution roughly doubled the spread relative to building blind, and screenshots widened it further.
  • C. Cost and variance: The cost-quality frontier was consistent under machine-probed behavioral scores and human-graded interface scores, with boot_check efficient and execution carrying a cost premium.The judgments remained separate because API behavior and human usability measure different things.

D. The Sight Boundary

Screenshots show their strongest descriptive benefits on visually visible mistakes, but the average advantage over a shell is modest, statistically unestablished, and not explained by screenshot usage alone.

  • D. The Sight Boundary: Visual and visual_no_shell exceeded execution on the calendar by +10.9 and +4.3 points, and on the dashboard by +2.9 and +5.7 points.These are mean human interface scores over assigned runs, with non-starters counted as zero.
  • D. The Sight Boundary: The pooled visual-versus-execution contrast was +6.9 points with permutation p = .041, but Holm correction changed the p-value to .083.The authors therefore characterize sight’s benefit on visually graded work as suggestive rather than established.
  • D. The Sight Boundary: Sight without a shell matched sight with a shell on both visual venues, isolating the visual channel rather than shell access as the relevant comparison.Figure 7 reports mean human interface scores across all six models, with individual model means shown in each bar.
  • D. The Sight Boundary: Screenshot gains were largest where mistakes involved visible placement or interaction, while measurement-only failures received no comparable benefit.The study reports that task-dependent visibility shaped average gains, but these patterns were descriptive and not statistically tested.
  • D. The Sight Boundary: The pooled benefit was driven largely by grok-4.3, which gained 42.9 points on the calendar with one screenshot and 35.6 points on the dashboard with none.Because the gains aligned with room for improvement rather than screenshot use, the study cannot credit them to models seeing and fixing their pages.
  • D. The Sight Boundary: Screenshot count did not track improvement: the largest gain came from a model taking one screenshot, while most heavy users remained near zero.Figure 8 plots screenshots taken horizontally and improvement over execution vertically; its pattern does not support a simple dose-response account.

E. Brownfield modification: a clean null

Verification tools showed about the same sensitivity when agents modified a small, fresh, well-structured codebase as when they built the same board from scratch. The result is a near-zero difference, but the confidence interval leaves smaller effects unresolved.

  • Evidence: 10.0 points on regression and 14.0 points on feature scoring were the modification-task execution gaps, versus 10.6 and 11.8 points for the matching greenfield build.The corresponding difference-in-differences was -0.7 points, with a 95% confidence interval from -13.7 to +12.7.
  • Finding: Modifying unfamiliar code showed about the same tool sensitivity as writing fresh code, not more.The study reports this as the pre-specified null comparison.
  • Interpretation: The wide interval rules out large differences in either direction but cannot exclude a smaller difference.The authors distinguish finding no difference from showing that none exists.
  • Scope: The brownfield starter was small, well structured, and freshly seeded, so the result may not extend to larger or messier codebases.This is the study's stated scope caveat for the null finding.
  • Integrity: Fingerprint checks supported that agents actually modified the supplied starter rather than discarding it and rebuilding from scratch.A rewrite was defined as changing or deleting more than half of the six starter files.
  • Boundary condition: On the log-explorer task, screenshots added nothing beyond the shell because its failures required running the application with substantial data rather than viewing a static image.The screenshot-over-shell contrast was -2.2 points, with interval [-12.5, +7.3].
  • Boundary condition: The log-explorer task produced 13 of 66 genuine launch failures, with 11 concentrated in the no-verification and static conditions.Its execution-versus-no-verification effect was +15.3 points, but it did not survive family correction.
  • Measurement: Human grading captured visible performance signs rather than timed speed measurements because the study lacked a tool for measuring the specified speed targets.The reported evidence therefore concerns freezes, blank regions, and an overgrown page.

G. Self-authored tests are not a substitute

Allowing agents to run only their own tests performed worse than providing the full execution shell. The shortfall reflected both fewer surviving builds and tests that validate an agent's own mistaken understanding.

  • Mechanism: The shortfall came partly from lower survival and partly from blind spots that an agent's own tests cannot expose when they encode the wrong task understanding.A test suite can pass while the underlying misunderstanding ships.
  • Result: The behavioral condition had 83% survival and a 71.2 functional score on seat-booking, versus 87.3 for execution on that task and 94.2 pooled across API tasks.This condition ran on 24 seat-booking runs and was the weakest configuration in the study.
  • Process: Behavioral runs used 369 thousand tokens and 22 steps on seat-booking, compared with 810 thousand tokens and 49 steps for shell runs, while writing similar amounts of code.The agents stopped early after writing tests and seeing them pass; no run was cut short by the harness or token budget.
  • Result: -16.1 points was the behavioral-minus-execution functional-score contrast, with a 95% interval of [-31.7, -1.7] and p = .043.The contrast was negative for five of six models, but it was a secondary finding from one task.
  • Failure example: A linter invocation failed because no lint script existed, and a later syntax error was never re-checked before finishing.The example distinguishes granting a tool from wiring it up, invoking it, and heeding its output.
  • Measurement: The study's hazard log and uniform static scanners measured related but non-identical defect conditions, so their totals differ by design.Both datasets were released separately.

VI. CONFIGURATION INTEGRITY AND MODEL

Configuration defects were partly reduced by execution tools, but model-specific implementation habits were more predictive of structural choices than tool condition. The evidence supports model signatures as observational patterns, not causal effects.

  • Defect mechanism: The boot probe nearly eliminated the data-directory bug because its single call exercised the failing step, although running the development path only reduced rather than eliminated the defect.Some execution builds checked only the production build and still shipped broken development configurations.
  • Defect rates: 169 of 1,116 builds, or 15%, contained the data-directory bug; rates fell from about 21–26% in no-verification and static conditions to 5.7% with the boot probe.Shell-condition rates were around 11–13%, while wiring problems affected 45 builds, or 4%.
  • Method: Cramér’s V measured the association between each categorical engineering choice and model or condition across ten extracted structural features.The features included project layout, module system, process orchestration, API calls, TypeScript, build tooling, tests, and README usage.
  • Model signatures: Seven of ten tracked structural choices were predicted far better by model than by tool condition.Whether an app shipped tests was the main condition-linked choice, because the behavioral condition required models to write tests before running them.
  • Examples: gpt-5.5 used a src-tree layout in 77% of runs and a bare frontend-build invocation in 94%, while grok-4.3 mixed root and nested manifests in 82%.These recurring choices were described as style preferences rather than mistakes.
  • Interpretation: Model signatures were observational because models were fixed comparison groups rather than randomized treatments, with imperfectly matched run counts.The authors therefore treat the patterns as descriptive rather than proven causes.

VII. LIMITATIONS

The study's visual and statistical conclusions are bounded by a passive screenshot channel, human-only grading for several tasks, limited independent model blocks, and incomplete reasoning continuity. These constraints limit what the results can establish beyond the tested settings.

  • Passive screenshot tool: The screenshot tool could render a URL but could not click, type, drag, hover, or resize, restricting it to passive visual verification.The agents repeatedly encountered this bound in their run logs.
  • Usage: 214 of 264 visual-condition runs took screenshots, totaling 1,326 calls to 260 distinct URLs, including 39% aimed outside the application root.These calls indicate effort spent reaching states the agent could not enter through interaction.
  • Passive screenshot tool: A non-interactive calendar defect survived visual verification because the page rendered correctly while ignoring clicks.The tool is structurally unable to detect faults that appear only through interaction.
  • Agent process: Reasoning content was not returned across turns, so reasoning continuity could not carry forward; shell conditions also ran longer than blind conditions.The limitation applies equally across conditions but bounds what the comparisons address.
  • Grading: Condition-blind human grades reproduced item verdicts at 98.8% in a 10% regrade, but this establishes stability rather than correctness.The pooled weighted kappa was 0.97, and survival calls agreed completely.
  • Grading: Three tasks lacked automated functional tests, allowing a visually correct but functionally empty dashboard to receive an overly generous rubric score.The human rubric was the only grade for the calendar, dashboard, and log explorer.
  • Statistical power: Statistical strength came from six different models rather than four or five replicates per cell, because replicates often share model-specific quirks.Within-model comparisons therefore cannot reach significance on their own and are descriptive.

F. Correction and null results

Across controlled builds, verification value separates into tiers governed by reach: boot checks address launch failures, shells address runtime behavior, and screenshots address visible defects. Some apparent effects are suggestive rather than established after multiple-comparison correction, while null results may still permit smaller effects.

  • Statistical caution: Two effects lose significance after correction for testing six effects, and a wide confidence interval around the brownfield null excludes only large effects.Surviving corrected effects are supported more strongly than uncorrected or imprecise null findings.
  • Survival: A boot probe removes nearly all launch failures caused by environment assumptions and source corruption, but cannot detect a broken frontend when the backend is healthy.One build in 192 still failed because the probe only checks whether the server starts.
  • Behavioral depth: Shell-equipped builds alone exposed behavioral failures involving concurrency, restart persistence, and ordering under load.These checks require actually running the application and distinguish behavioral depth from cheaper verification conditions.
  • Visual quality: Screenshots help most on visually observable tasks, add nothing when failures are measurable but invisible, and show only a modest uncorrected advantage over a shell.The visual effect does not survive stricter multiple-comparison correction, and its mechanism remains unresolved.
  • Practical implications: The practical recommendation is to use a cheap boot probe first, add a shell for behavioral depth, and treat screenshots as task-dependent rather than interaction testing.The shell costs 2.35 times as much as the no-tools condition, while screenshots are most relevant to visually hard work.
  • Interpretation: The study’s central rule is reach: each tool improves artifacts only where it can exercise the failure mode, while a linter may inspect nothing in a quarter of runs.A single pass rate cannot distinguish launch failure, near-correct behavior, or a visually correct but nonresponsive page.

APPENDIX A

The appendix preserves the agent interface texts and the study’s reproducibility archive. It records byte-exact prompts and identifies the archived data and reproduction materials.

  • Agent Interface: The base prompt and universal environment briefing are reproduced byte-exact, with hashes verified against every run manifest.The base prompt uses a workspace placeholder and instructs the agent to implement the user’s task with only provided tools.
  • Reproducibility: The full dataset and reproduction recipes are archived under Zenodo DOI 10.5281/zenodo.21961590 and browsable in the linked GitHub repository.The archive includes a SHA256-sealed dataset snapshot and instructions for regenerating analyses, recomputing statistics, and regrading artifacts.
Loading 2608.28795v1…