Source-linked AI summary
From Blind Edits to Verified Repair: Building Trustworthy User-Side LLM Agents for Web Accessibility
Lily Bundgaard Wanscher, Markus Heidemann Lorensen, Mohammed Ammad Shafiq, Mahyar Tourchi Moghaddam, Mina Alipour
TL;DR
Web accessibility failures remain widespread, motivating privacy-preserving user-side LLM agents that can adapt pages without site-author cooperation. This paper builds and evaluates such an agent, finding that unverified repairs improve and regress pages at similar rates, while verification reliably detects seeded violations and rejects harmful edits.
Problem
User-side accessibility adaptation is a promising complement to author remediation, but evidence for privacy-preserving local LLM agents repairing live webpages remains limited.
Method
The paper combines a dual-condition live-site evaluation with a trilingual seeded benchmark and an audit-inject-verify loop that accepts edits only when violations decrease.
Results
Unverified generation produced 24 improvements and 20 regressions across 100 trials, while the verified instrument detected 57/57 seeded violations and rejected 126/126 harmful candidates.
Takeaways & Limitations
Trustworthy accessibility agents require verification of every edit against the rendered page rather than relying on unverified model generation.
Takeaways & Limitations
The study tested only home pages and did not include target-population user evaluations, so its results do not establish experienced accessibility.
Abstract
from arXiv · showhide
Assistive agents that adapt web pages on the user's side, at the moment of browsing, could reach the accessibility failures that site authors leave unfixed, and large language models make such agents newly plausible. We contribute three building blocks toward that goal. The first is a complete, privacy-preserving browser agent: a Chrome extension that extracts a page's style sheets, condenses them to fit a local model's context window, asks the model for additive CSS addressing 18 metrics from WCAG and the W3C cognitive accessibility guidance, and injects the result reversibly into the live page. The second is a dual-condition protocol that measures harm as carefully as benefit, applied to six small open-weight models (7B to 14B) on ten violation-rich and ten highly accessible live sites. The diagnosis is sobering but precise: unverified generation improved and regressed pages at similar rates (24 improvements against 20 regressions across the 100 trials of the five models that produced injectable CSS), fixing typography while breaking perception-dependent properties. The third answers the diagnosis: a verified repair instrument pairing a trilingual seeded-violation benchmark with an audit-inject-verify loop that accepts a change only if violations strictly decrease, so regression on the automated checks is impossible by construction. In a real browser the instrument detects 57 of 57 seeded violations with no false positives and rejects 126 of 126 adversarially harmful candidates. All code, prompts, benchmark materials, aggregate data, and validation logs are released.
1 Introduction
Web accessibility failures remain widespread, motivating a privacy-preserving browser agent that uses local language models to inject additive CSS without site cooperation. Across live-site trials, unverified repair produced both improvements and measurable regressions, prompting a verified-repair instrument.
- Motivation: 95.9% of the top one million home pages had WCAG 2 conformance failures, averaging 56.1 distinct errors per page; low-contrast text appeared on 83.9%.The cited analysis reports that accessibility has not improved despite mature guidelines, and suggests AI-assisted code generation may be widening the gap.
- Motivation: A user-side browser strategy can repair pages at consumption time without requiring cooperation from site authors, and large language models revive this approach by rewriting page style sheets.The paper situates this strategy in prior work on transcoding proxies, collaborative scripting, and community metadata authoring.
- System and evaluation: The system is a Chrome extension that condenses active-tab styling, prompts a locally hosted model for additive CSS addressing 18 WCAG and cognitive-accessibility metrics, and injects recovered valid CSS.The intervention is designed as a minimal but complete browser-based instantiation for consumer hardware.
- System and evaluation: Six open-weight models, ranging from 7B to 14B parameters, were evaluated on 20 live websites—10 violation-rich and 10 accessible controls—using scores from all 18 metrics before and after intervention.The accessible sites served as a do-no-harm control condition.
- Findings and contributions: 24 of 100 trials improved a site while 20 made one measurably worse; the sixth model produced no extractable CSS in any of its 20 trials, and no test approached significance.These results concern the five models that produced injectable CSS and show why the paper reports unverified repair as instructive rather than encouraging.
- Findings and contributions: The paper contributes a privacy-preserving open-source browser agent, a dual-condition 18-metric protocol across six models, and a validated verified-repair instrument.The replication package includes the extension source, prompts, rubric, benchmark, harness, summary data, and analysis scripts.
2 Related Work
Prior work spans web accessibility conformance, user-side page transformation, and LLM-based remediation, but automated evaluation remains incomplete and existing LLM studies often emphasize text-centric issues. This study instead combines local small models, browser-side CSS repair, privacy, and an accessible control condition that measures harm as well as benefit.
- WCAG is the de facto web conformance target, while W3C cognitive-accessibility guidance adds concerns such as content organization and clear typography.Automated checkers catch only a fraction of real barriers, and relying solely on them harms evaluation quality.
- Earlier user-side systems adapted pages through narrow scripts, transformations, or external metadata, achieving reliability by encoding specific fixes.Examples include transformation-based accessibility, Accessmonkey, and Social Accessibility.
- Most LLM accessibility studies target text-centric issues, use large commercial models, and rarely evaluate with disabled users.This study differs through user-side live-browser intervention, locally running small open-weight models, CSS-focused repair, and an accessible control condition measuring harm.
- 37 of 39 WAVEdetected violations were repaired by ChatGPT on two sites, but at 88-site scale only about 70% proved fixable, with systematic contrast and perceivability failures.Related work also reports generative pipelines outperforming classical checkers, while some LLM suggestions alter page semantics while claiming to repair them.
3 The Accessibility Agent
The accessibility agent is a privacy-preserving Manifest V3 Chrome extension that locally generates additive, reversible CSS from condensed page styles. Because models inconsistently return parseable raw CSS, the agent recovers and validates candidate styles before injecting them.
- Design goals: The agent uses local inference, requires no site-author cooperation, applies additive reversible modifications, and caps context at 10,000 tokens for consumer hardware.It pairs a Manifest V3 Chrome extension with an Ollama server on the same machine and assumes a 16 GB GPU.
- Page-style extraction: The extension extracts CSS from document.styleSheets, fetching cross-origin sheets by URL when direct enumeration raises a security error.It runs through chrome.scripting in the focused tab and collects the text of every CSS rule.
- Context condensation: Two-pass condensation filters token-expensive nonvisual declarations, then ranks surviving rules by selector reach, high-impact properties, and !important usage.The budget reserves 40% of the min(model context, 10,000)-token window for the response and admits roughly 20,000 characters at the cap.
- CSS generation: The user prompt asks for additive CSS improvements across 18 accessibility metrics, including quantitative WCAG thresholds where applicable.The system prompt explicitly requires raw CSS beginning with a selector and ending with a closing brace.
- Recovery and validation: No tested model consistently followed the raw-CSS instruction, so the agent recovers CSS from multiple response formats and rejects candidates that fail parsing or basic structural checks.Valid CSSStyleSheet objects are appended to the document’s adopted style sheets.
4 Study Design
The study evaluated a privacy-preserving CSS-editing agent across twenty live websites, six open-weight models, and an 18-item accessibility rubric. Trials compared before-and-after violation counts, categorized outcomes, and supplemented planned tests with exact sign tests suited to sparse integer data.
- Websites and conditions: Twenty live websites were divided into ten violation-rich commercial sites and ten highly accessible control sites.The sample condition targeted sites where improvement was possible in principle, while the control condition contained sites judged to have negligible violations.
- Evaluation rubric: The evaluation rubric contained 18 CSS-addressable metrics drawn from WCAG 2.1, WCAG 2.2, and W3C COGA guidance.The rubric covered perception, typography, and layout and reflow; HTML or ARIA changes were out of scope.
- Models: Six open-weight instruction-following models ranged from 7B to 14B parameters across smaller and medium model groups.The models were selected from Ollama to vary vendor and architecture family, with availability and speed on the study hardware prioritized over expected capability.
- Trial protocol: Each trial captured before-and-after screenshots after CSS injection, discarding unrecoverable responses and recording no change after ten consecutive invalid responses.The protocol reran trials when valid CSS could not be recovered before applying the ten-response stopping rule.
- Outcome measurement: Outcomes were defined by the violation difference, calculated as after minus before, and categorized as improvement, regression, or neutral edit.Negative violation differences indicated improvement; improvement required at least one violation removed with none added on balance, while regression meant net violations added.
- Statistical analysis: The analysis paired planned two-tailed one-sample t-tests at α=.05 with exact two-sided binomial sign tests because violation differences were small integers with many zeros at n=10.The sign tests compared improvements against regressions per model, per condition, and pooled.
5 Results
Across five models that produced injectable CSS, unverified edits yielded similarly frequent improvements and regressions, with small, often null changes in violation counts. Statistical tests found no supported directional benefit for violation-rich sites or evidence that harm to accessible sites was rare, while StarCoder2 7B failed in every trial.
- Model availability: StarCoder2 7B produced no injectable CSS in any of its 20 planned website trials.Its recovery cascade never extracted valid CSS, so those trials yielded no outcome data beyond failure.
- Outcome categories: Across 100 trials from five producing models, control sites had 14 improvements and 14 regressions, versus 10 improvements and 6 regressions on sample sites.These corresponded to 28% improvement and regression rates on controls and 20% versus 12% on sample sites.
- Outcome categories: 56 of 100 trials left net violation counts unchanged, with per-site differences ranging from −2 to +4 on controls and −2 to +2 on sample sites.The worst outcome added four violations to one control page, while no trial removed more than two.
- Model differences: Model efficacy patterns diverged: Qwen2.5-Coder and DeepSeek-R1 each improved five of ten control sites, whereas Gemma 3 improved three sample sites but only one control site.Qwen had one control regression, DeepSeek-R1 had three, and both were largely inert on sample sites; Gemma 3 produced three control regressions.
- Statistical tests: No statistical test supported a directional effect: pooled control improvements equaled regressions (14 versus 14, p=1.0), while sample-site outcomes were indistinguishable (10 versus 6, p=.454).Per-model t tests ranged from .62 to 1.0, sign tests stayed at or above .219, and improvement rates between conditions did not differ (Fisher exact p=.483).
- Statistical tests: The study could neither conclude that the intervention improves violation-rich sites nor that harm to accessible sites is rare.The apparent difference in regression rates between conditions was only an exploratory, nonsignificant trend (p=.078).
6 Discussion
The discussion finds user-side accessibility repair technically feasible but not yet trustworthy, because blind, partially observed CSS edits can cause regressions and violate output constraints. It therefore frames deployment around perceptual grounding, verification, reversibility, consent, and context awareness.
- Feasibility and trust: Zero-cooperation, privacy-preserving intervention is feasible through extraction, condensation, local generation, and additive injection, but manual triggering remains necessary while actions are untrustworthy.The pipeline could run in the background from a service worker, yet autonomy is justified only when actions can be trusted.
- Failure mechanisms: Globally settable typography yields to broad declarations, whereas rendered properties such as contrast, reflow, target size, and focus visibility require perceptual grounding.The model cannot directly see the rendered page, so it must infer relationships across foregrounds, backgrounds, cascade, and inheritance.
- Failure mechanisms: Truncated context makes locally sensible additive CSS globally harmful by hiding selectors and specificity, with regressions concentrating on accessible control sites.Budget packing discards rules, while tuned sites contain more conformant elements that can be broken.
- Safety requirements: Every model breached the raw-CSS output contract, and one met with total failure, requiring deployed agents to parse, validate, and sandbox model output.Model output must be treated as untrusted input rather than executed verbatim.
- Evaluation limits: 18 metrics per capture limited the study to 120 trials, while rubric conformance bounded rather than measured lived accessibility and user testing was absent.An automated checker is needed as both a runtime guardrail and a scalable evaluation component.
- Design requirements: Assistive agents should pair LLM generation with rendered evidence or deterministic WCAG computations, verification loops, reversible changes, human consent, and locale-aware evaluation.Targeting individual violations can reduce context demands; CSS-level operation is language-independent, but HTML, ARIA, and content require locale and cultural awareness.
7 From Diagnosis to Design: A Verified Repair Instrument
The verified repair instrument combines a seeded benchmark, automated auditing, and an audit-inject-verify loop that makes regression on checked violations impossible by construction. Validated in real Chromium, it rejects harmful candidates, supports vision-model requests, and enables unattended benchmark evaluation.
- Instrument design: The instrument pairs exact-ground-truth seeded pages with an automated auditor and an acceptance rule requiring strictly fewer violations without new violation types.All components ran in a real Chromium instance, and the accept rule rolls back unsuccessful repairs.
- Benchmark: 42 locally served pages span English, Danish, and German, including seeded violations and clean controls.Each locale contributes ten violation-bearing pages and four clean controls; seeds cover eight CSS-level types.
- Validation: 126 of 126 harmful candidates were rejected by the verification loop.A mixed generator cycling fixes, harm, and invalid prose reached the same end state through retries.
- Verification loop: The loop audits, requests CSS, validates parsing, injects, re-audits, and retries up to three attempts before rollback.Retries include the concrete failing checks, while acceptance requires strictly decreased total violations and no new violation type.
- Deployment and evaluation: The instrument supports screenshot-attached requests for vision-capable models and reduces full-benchmark evaluation to one command plus inference-bound hours.Human spot-checks remain necessary because residual risk is limited to regressions the automated checks cannot see.
8 Limitations and Threats to Validity
The evaluation is limited by a small trial count and bounded sensitivity, while scoring relied on one evaluator and included judgment-dependent rubric items without reliability cross-checks.
- 120 trials were planned, including 100 completed trials for the five models that produced injectable CSS.The design used ten sites per condition per model.
- Sensitivity was bounded at roughly |d| = 1.0, and one scoring error could shift observed rates by several points.
- A single evaluator scored all results without a second rater or reliability statistic, while several rubric items required judgment.The judgment-dependent items included content arrangement, hierarchy, and color symbolism.
9 Future Directions
Future work should extend repair beyond CSS, manage the resulting token costs with targeted diffs, and evaluate text and vision models through the verification instrument.
- Structural repair: Extending repair from CSS to HTML and ARIA attributes would address structural barriers such as accessible names and roles for screen reader users.The architecture needs no change beyond revising extraction and injection targets.
- Efficiency: The expanded repair surface would worsen token economics, increasing the need for targeted diffs.
- Model evaluation: The immediate evaluation step is to run both text and vision study models through the Section 7 instrument.
10 Conclusion
The paper delivers a released user-side accessibility agent, an honest harm-aware measurement protocol, and a verified repair component. Unverified generation improved and regressed live pages at similar rates, revealing the need for verification.
- The released agent demonstrates that small local LLMs can repair web accessibility from the user’s side.The conclusion describes the agent as working and characterizes this as solved engineering, released in full.
- A harm-aware protocol found that unverified generation improved and regressed live pages at similar rates.The system fixed properties addressable by a global declaration while breaking properties it could not see.
- The paper builds the missing verified repair component needed to address the limits of unverified generation.