Agentic Web Accessibility Auditing: A Criterion-Specific Framework for Translating WCAG Requirements into Assessments

1 Introduction

• Rule-based checks can identify some violations, but image meaning and keyboard reachability, activation, and exit require different evidence.A tool may provide partial coverage of a criterion without establishing that it is satisfied in every relevant state.Source: §1 · pp. 1–2
• The framework assigns a worker agent to each targeted success criterion, with shared tools and control but criterion-specific evidence requirements and decision instructions.Each worker returns a structured assessment, making the relationship between a requirement, an action, and a reported finding explicit and enabling inspection of failures by assigned requirement.Source: §1 · p. 2
• The study instantiates 40 workers, including 39 of the 50 WCAG 2.1 Level A and AA criteria and one additional WCAG 2.2 criterion; positive reference cases cover 15 criteria, so violation-detection performance cannot be established.Source: §1 · p. 2
• The evaluation derives positive labels from Library Accessibility Alliance audit reports and treats the absence of a listed finding as an inferred negative.This construction supports comparison of reported positives and negatives, but does not make the audit reports exhaustive assessments of every saved page state.Source: §1 · p. 2
• Reference workers recover 0.86 of positives, versus 0.36 for axe-core and 0.67 for an uncued batch vision-language model; their respective precisions are 0.56, 0.90, and 0.61.The primary analysis uses 250 page–criterion records from 24 pages across 11 scholarly platforms, including 78 positive and 172 inferred-negative labels.Source: §1 · p. 2
• The paper contributes a criterion-specific architecture, a translation of requirements into evidence and decision instructions, and an evaluation corpus with reanalysis of agreement, abstention, and resource use.Criterion-level comparisons and sensitivities address development exposure and report provenance, while professional-auditing effects remain to be evaluated.Source: §1 · p. 3

2.1 Accessibility evaluation, conformance, and professional practice

• Professional evaluations distinguish sampled audits from whole-site conformance, so reports do not guarantee every criterion was exhaustively tested on every page.A positive finding records an identified problem, while silence may reflect sampling, report purpose, or page state and requires an additional assumption.Source: §2.1 · p. 3
• Inspectable evidence can aid review after execution, but unclear locations or transient interface states can make verification costly even when findings are inexpensive to generate.The framework distinguishes computational cost from human review cost and uses page–criterion assessments rather than site-level conformance claims.Source: §2.1 · pp. 3–4

2.2 Rule-based and interaction-based accessibility testing

• The framework records operations and reports an assessment for an assigned WCAG success criterion on a particular page.This criterion-level evidence relationship differs from assuming that results transfer unchanged across platforms.Source: §2.2 · p. 4
• TaskAudit begins with generated mobile tasks, whereas these workers begin with a named criterion and gather relevant evidence on a rendered web snapshot.The configured axe execution cannot represent every deterministic test, so its comparison does not establish that all rule-based methods lack worker capabilities.Source: §2.2 · p. 4

2.3 Language and vision-language models for auditing

• Criterion-specific executable operations permit further evidence collection, and the study examines whether they accompany more detected reference cases or more false positives.The evaluation uses a fixed archival sample, leaving adaptive page selection outside its empirical claims.Source: §2.3 · pp. 4–5
• Vision-language conditions receive a screenshot and accessibility-tree information, including browser-exposed roles, names, states, and relationships, rather than only pixels or an observed interaction sequence.The tree remains distinct from observed keyboard or screen-reader interactions.Source: §2.3 · p. 5

2.4 Agent architecture and human oversight

• The evaluation tests a configured architecture in which criterion guidance and tool access change together in some comparisons, measuring predictions and execution records rather than demonstrated benefits to auditor decisions.The cited human–AI literature motivates testable implications for auditor review, not a measured decision-making benefit here.Source: §2.4 · p. 5

3.1 From a success criterion to an executable assessment

• Each worker scopes one WCAG criterion by specifying what to inspect, how to observe it, and when observations support a result, without changing the requirement.Criterion-specific instructions state what additional evidence is needed before reporting a finding.Source: §3.1 · pp. 5–6

3.2 Shared execution and criterion-specific tools

• Workers alternate reasoning and browser actions, combining DOM, rendered, and keyboard observations through shared tools selected for each criterion.Operations include document and accessibility-tree extraction, screenshots, keyboard and pointer actions, style inspection, contrast measurement, and axe checks.Source: §3.2 · pp. 6–7
• A declared tool list is guidance, not an enforced capability boundary, and a tool’s structured summary is not independently verified evidence.Reviewing a disputed finding may require examining the underlying operations rather than relying on the summary alone.Source: §3.2 · p. 8

3.3 Authoring and implementation coverage

• Authoring links criterion instructions to behavior; the implementation includes 40 workers: 39 for WCAG 2.1 A/AA criteria and one for WCAG 2.2 criterion 2.5.7.The inventory omits 11 WCAG 2.1 A/AA criteria: 2.4.5, 2.4.7, 2.5.2, 2.5.4, 3.2.1–3.2.4, 3.3.1, 3.3.3, and 4.1.3.Source: §3.3 · pp. 8–9
• Decision guidance requires contextual checks: empty alternative text may suit decoration, while repeated focus requires checking exit actions and legitimate widget behavior.The passages distinguish criterion text, tool output, and verdict as different parts of an assessment.Source: §3.3 · p. 9
• Only 15 of the 40 implemented criteria have positive reference cases in the primary benchmark, so recall cannot be estimated for the other 25.The WCAG 2.2 worker’s five records have inferred-negative labels; the authors distinguish worker implementation from evidence that it detects violations.Source: §3.3 · p. 9

3.4 Execution evidence and a worked example

• Execution records capture tool calls, arguments, observations, timestamps, and results; reviewers must assess whether supporting observations justify the worker’s interpretation.A model-generated explanation alone can overstate what the execution established.Source: §3.4 · p. 9
• After selected axe checks found no violations, the worker gathered image and DOM context, captured an execution-linked screenshot, and judged some empty-alt images informative and others decorative.The archived screenshot links the displayed page region to the recorded operation sequence.Source: §3.4 · p. 9, Figure 2 · p. 10
• The row scored true positive because both worker and audit were positive; criterion-level agreement need not match issue location or explanation, and the example does not independently validate the interpretation.The score does not require the worker to identify the same image as the report.Source: §3.4 · pp. 9–10

4.1 Reference labels and sampling

• A missing report finding is treated as an inferred negative, not as the result of an independent pass/fail audit of that page–criterion pair.Positive-case sampling raises their share for missed-finding analysis, so precision reflects agreement with this labelled sample rather than arbitrary websites.Source: §4.1 · pp. 10–11
• 250 records cover 24 pages from 11 scholarly platforms, with 78 reported positives and 172 inferred negatives across 40 criteria.The corpus reflects available reports and intended captures, not random sampling of the web.Source: §4.1 · pp. 10–11

4.2 Saved documents and experimental provenance

• Each evaluation record maps a success criterion to a saved HTML capture, which permits DOM, accessibility-tree, appearance, and some interaction inspection.A capture may not reproduce production authentication, remote requests, timers, or state transitions, and the original audit may have examined another version.Source: §4.2 · p. 11

4.3 Development exposure and sensitivity populations

• The diagnostic files touch 27 evaluation rows across 19 pages, so the remaining 223 rows are not treated as an independent held-out set.A stricter sensitivity analysis excludes every page represented in the diagnostics, leaving 55 rows on five pages, including 15 positives; it does not establish prospective generalization.Source: §4.3 · pp. 11–12
• Additional sensitivity analyses separate vendors, exclude axe-associated reference findings, exclude rows requiring unavailable dynamic behavior, and omit each platform in turn.Each comparison uses the same population across conditions and reports its positive denominator.Source: §4.3 · p. 12

5.1 Rule-based and vision-language assessment

• The batch VLM uses three calls per page, dividing WCAG criteria among Perceivable, Operable, and Understandable/Robust stages and matching reported identifiers to evaluation rows.Uncued calls receive the screenshot and accessibility tree; cued calls also receive criterion-related counts from the captured element data.Source: §5.1 · p. 12
• The per-criterion VLM assesses each page–criterion pair using its normative statement, page representation, criterion-specific heuristic cue, and an option to abstain.The cue is a count from simple rules over captured element data; these heuristics flag patterns for inspection rather than establish violations.Source: §5.1 · p. 12

5.2 Worker and general-purpose coding-agent configurations

• The worker and coding-agent comparisons vary in tools, instructions, or evidence processing, so differences between products cannot be attributed solely to a model, prompt, or tool-use strategy.General-purpose agents are tested with a brief audit request and a parity request enumerating the targeted criteria.Source: §5.2 · p. 13

5.3 Metrics and resource accounting

• Precision is the proportion of positive predictions with positive reference labels, while recall divides true positives by all reference-positive rows; abstentions and errors do not recover positives.The confusion tables keep abstentions and execution errors separate from positive and negative decisions.Source: §5.3 · p. 13
• Costs use complete 250-row workload totals and are paired with accuracy from the same run and population; per-row costs allocate shared page-level calls rather than represent separate requests.Token estimates, API-equivalent subscription estimates, processing spans, and elapsed batch time are not treated as interchangeable quantities.Source: §5.3 · pp. 13–14

6.1 Overall agreement and implementation coverage

• Workers recovered 0.86 of reference positives at precision 0.56, versus axe-core’s recall 0.36 and precision 0.90 and the uncued batch VLM’s recall 0.67 and precision 0.61.F1 was 0.68 for workers, 0.64 for the uncued VLM, and 0.69 for the cued VLM.Source: §6.1 · p. 14
• Enumerating criteria raised Claude Code’s recall from 0.60 to 0.77 and Codex’s from 0.32 to 0.54, with additional false positives in both configurations.The comparison did not equalize system instructions, tools, or models; Claude Code used a different recorded model from the workers.Source: §6.1 · p. 14

6.2 Criterion-level variation and abstention

• Workers recovered seven of eight Keyboard cases and both No Keyboard Trap cases, while the uncued batch VLM recovered none; for Focus Order, it recovered four of seven versus six for workers.For Non-text Content, the per-criterion VLM recovered seven of eight cases versus six for workers; for Contrast (Minimum), the uncued VLM recovered four of five versus three for workers. Focus Order shows noninteractive source and layout evidence can support detection without reproducing complete behavior.Source: §6.2 · p. 15
• The per-criterion VLM abstained on 92 of 250 rows (36.8%), including 27 positive-labelled rows; its full-population recall was 0.59 versus conditional recall of 0.90 on committed rows.Committed rows included 51 positives, of which the model recovered 46; on that same subset, cued batch VLM recovered 45, while uncued batch VLM and workers each recovered 42. The VLM also declined all seven No Keyboard Trap rows, including two positives; abstention or commitment alone does not establish evidence sufficiency.Source: §6.2 · p. 16

6.3 Variation across recorded model configurations

• Across seven archived batch VLM configurations, primary recall ranged from 0.47 to 0.77, none identified either No Keyboard Trap positive, and Keyboard true positives ranged from zero to four of eight.Instrumented workers using Gemini 3.1 Pro, Claude Sonnet 5, and GPT-5.5 recovered 60, 62, and 66 of 78 positives; a GPT-5.5 repeat recovered 69 and agreed with the first pass on 241 verdicts (96.4%).Source: §6.3 · p. 16

6.4 Costs attached to their own runs

• Batch VLM estimates were USD 6.38 uncued and USD 7.29 cued, versus USD 14.98 per-criterion; September 2026 worker estimates ranged from USD 36.65 to USD 64.19.These costs pair with each complete run’s accuracy, but describe different call allocations; the estimates are not uniform billing measurements, and no cost was assigned to the reference worker run because matching usage was unavailable.Source: §6.4 · pp. 17–18, Table 6 · p. 18, Figure 6 · p. 19

6.5 Complementarity and sensitivity analyses

• The retrospective union of worker and uncued VLM predictions recovered 74 of 78 positives (precision 0.54, recall 0.95), while their intersection yielded 45 true and 23 false positives (precision 0.66, recall 0.58).The union adds positive predictions absent from reports, while requiring agreement reduces both detections and false positives; escalation and adjudication were not evaluated.Source: §6.5 · p. 19, Figure 7 · p. 20
• After excluding development-exposed pages, workers recovered 14 of 15 positives, the uncued VLM ten, and axe six; this 55-row subset does not establish transfer to future deployments.Shared instructions revised on excluded pages may still affect retained pages.Source: §6.5 · p. 19, Table 7 · p. 20
• The analysis checks whether results concentrate on documented development pages rather than using a prospectively separated evaluation.Source: §6.5 · p. 20
• Worker recall is higher and precision lower than axe in both vendor-specific report subsets.Source: §6.5 · p. 20
• Excluding axe-provenance rows or dynamic-only flags yields distinct positive-case populations, and neither replaces the headline based on better performance.Dynamic-only flags include cases where required behavior and snapshot capabilities mismatch; the populations answer different questions.Source: §6.5 · p. 20
• Historical worker recall ranges from 0.84 to 0.90 across leave-one-platform-out analyses, versus 0.64 to 0.70 for the uncued VLM and 0.33 to 0.37 for axe.These ranges describe sensitivity to platform composition, not confidence intervals or significance tests.Source: §6.5 · p. 20

Diagnostic flags and disagreement causes

• Archived diagnostic flags are not independently confirmed label corrections, and their populations differ from worker false-positive categories.A candidate could reflect an omitted finding, a changed page, an incorrect interpretation, or a criterion mismatch.Source: §7 · p. 21

Candidate findings and label uncertainty

• 27 of the reference workers’ 53 positive predictions on negative-labelled rows carry candidate-discovery flags, but the candidates lack independent adjudication.The authors retain the original labels and treat proposed amendments as claims requiring review.Source: §7 · p. 21
• Accepting all 27 flagged candidates as additional positives would yield historical worker precision of 0.78 and recall of 0.90, a conditional sensitivity rather than validated performance.The calculation changes the recall denominator from 78 to 105 positive labels.Source: §7 · p. 21

Diagnostic flags and disagreement causes

• Aggregate confusion counts cannot establish how often individual error mechanisms occur, so the records do not support a validated taxonomy of error causes.The authors avoid assigning all false positives to ambiguous judgments or all false negatives to inadequate probing.Source: §7 · p. 21

Interpreting keyboard evidence

• Focus retention alone cannot validate a No Keyboard Trap violation; a convincing trace must show attempted exit actions and resulting focus behavior.A selected radio option can legitimately retain focus, while the criterion concerns whether appropriate keyboard actions can move focus away.Source: §7 · p. 21

8.1 Matching assessment methods to evidence requirements

• Workers recover more keyboard reference cases than noninteractive VLM configurations, while some visual or semantic criteria favor a VLM condition.This supports criterion-oriented assessment, distinguishing evidence requirements from an absolute classification of an entire criterion.Source: §8.1 · p. 21
• Abstentions can prevent unsupported verdicts, but declined rows remain outstanding work and should be communicated with the missing observation and next inspection.A generic confidence label would not identify what remains unresolved.Source: §8.1 · p. 22

8.2 What report-derived benchmarks can establish

• Accepting a model’s disputed findings as new ground truth would weaken the independence of the evaluation; candidate amendments should remain claims requiring adjudication.Candidate-discovery sensitivity illustrates how label construction can affect measured precision.Source: §8.2 · p. 22
• Report-derived labels differ in assessment object and granularity: a criterion-level label can aggregate distinct problems, and report absence is not an exhaustive negative judgment.Future datasets could preserve audited page state, implicated elements or interaction sequences, and checked criteria without violations.Source: §8.2 · p. 22
• The manifest and row-level records provide part of the structure for evaluating whether an issue exists, where it occurs, and whether observations substantiate it.The authors suggest combining report-derived cases with mutation-based tests to distinguish missed seeded defects from uncertain natural-case labels.Source: §8.2 · p. 22

8.3 Allocating automated effort within an audit

• The proposed tiered workflow pairs deterministic checks with model-based assessment and targeted browser operations for unresolved cases, but its allocation rule remains unevaluated.The study does not measure human review or remediation costs; subsequent evaluation should include review time, adjudication accuracy, reproducibility, and action effort.Source: §8.3 · pp. 22–23
• Combining methods can recover more reference positives but creates more findings for review, while requiring agreement discards cases found by only one method.A routing policy must account for these consequences and correlated outputs, and its ordering requires evaluation.Source: §8.3 · p. 23

8.4 Inspectable evidence for human auditors

• Worker records could connect criteria to operations and observations, making supporting evidence inspectable rather than requiring acceptance of the explanation alone.Whether the current trace format helps professionals reproduce and assess findings efficiently remains open.Source: §8.4 · p. 23

Population and outcome scope

• The 24 pages represent scholarly platforms rather than a representative sample of web applications, and only 15 criteria have positive cases.Page–criterion scoring measures whether a criterion is flagged, not whether every reported element, explanation, or proposed cause is correct.Source: §9 · pp. 23–24
• The archival comparison does not reconstruct every historical execution setting or missing usage measurement, and tool access, instructions, prompts, and underlying models can differ.These differences prevent causal isolation of browser interaction or prompt granularity.Source: §9 · p. 24

Generalization and repeatability

• Development exposure and clustering limit generalization; the whole-page exclusion sensitivity is small and retrospective, so prospective separation would better test transfer.Diagnostic files touch most evaluated pages, and shared changes can affect untouched rows.Source: §9 · p. 24
• Repeated execution covers only two passes of one September 2026 worker configuration, while logged prices and usage estimates describe archived workloads rather than stable deployment costs.The study also does not measure human review or remediation costs.Source: §9 · p. 24

Environment and future evaluation

• Saved documents may differ from production services requiring authentication, network state, or multi-page processes, and evaluating practical consequences requires auditors and people with relevant access requirements.Future evaluation should include issue-level matching, direct evidence validation, explicit negative judgments, and user-facing outcomes without assuming progress on one outcome establishes another.Source: §9 · p. 24

• The framework combines shared browser tools with criterion-specific instructions to produce accessibility assessments.Source: §10 · p. 24
• On the report-derived benchmark, reference workers recover more positive labels than axe-core and an uncued batch VLM, but with lower precision.Source: §10 · p. 24
• Detection patterns vary across criteria, motivating assessment procedures matched to the evidence each requirement needs.Source: §10 · p. 24
• It preserves operations and observations, making the basis of each judgment available for inspection.Source: §10 · p. 24
• Professional-auditing benefits remain unevaluated, while criterion-specific evidence collection and inspectable execution records offer directions for developing assessment systems.Source: §10 · p. 24