AXNav: Replaying Accessibility Tests from Natural Language

1 INTRODUCTION

• Despite scanners, linters, and test automation, most accessibility testing remains manual because automated tools can be brittle, absent, or produce false positives.Manual testing can also reveal issues automated techniques alone cannot detect.Source: §1 · p. 2
• Manual accessibility testing is costly and hard to scale, and its scope can force testers to narrow tests, potentially letting bugs slip through and leaving instructions outdated.Previously recorded tests can require updates when UI or navigation flows change.Source: §1 · p. 2
• AXNav applies LLMs to accessibility testing and control of accessibility services and settings, which the authors identify as an unexplored use of LLMs.The work draws motivation from LLM use in recreating bug reports, testing GUIs, and automating web tasks [22], [36], [47].Source: §1 · p. 2
• AXNav turns manual accessibility instructions into replayable, navigable videos using an LLM and pixel-based UI element detection, with heuristics flagging potential issues.The system replays tasks through accessibility services so testers can visualize and hear the interaction.Source: §1 · p. 2
• The paper reports a formative study with 6 professional testers, a natural-language replay system, and a 10-tester study of its place in manual workflows.The system combines an LLM with pixel-based UI detection and flags issues such as non-increasing text and navigation loops.Source: §1 · p. 2

2.1 Large Language Models and UI interaction

• AXNav’s planner combines or removes modules from its multi-agent architecture to reduce LLM turns, cost, and latency for testing tasks.The design is loosely based on ResponsibleTA [58].Source: §2.1 · p. 2
• AXNav interprets instructions ranging from detailed steps to unconstrained goals, can adapt plans when interfaces change or errors occur, and does not rely on prior app knowledge.Unlike some approaches, it supports iOS apps and complex multi-step tasks.Source: §2.1 · pp. 2–3
• AXNav targets accessibility testing and accessibility-feature interaction, which the reviewed UI-navigation works had not addressed.The authors frame this as the core focus of AXNav’s contribution.Source: §2.1 · p. 3

2.2 Accessibility Testing Tools

• Accessibility scanners and linters can check properties statically or at runtime, but runtime tools must be activated on each screen being tested.Manual testing remains common, and testing through accessibility services can reveal issues scanners alone cannot reveal.Source: §2.2 · p. 3
• Random app crawlers may miss end-user paths and cannot assess navigation through accessibility services.They may therefore fail to evaluate whether app navigation paths are fully accessible.Source: §2.2 · p. 3
• Record-and-replay accessibility tools can require existing GUI tests or become brittle as interfaces evolve, while AXNav uses natural-language instructions and adapts to changed UI context.Latte converts GUI navigation tests; A11yPuppetry records and replays flows through accessibility services.Source: §2.2 · p. 3
• AXNav flags a subset of potential issues during replay to aid manual QA; it is not designed for holistic accessibility audits or reporting.The architecture could be extended to run audits at each replay step, but that is outside this work’s focus.Source: §2.2 · p. 3

3 FORMATIVE INTERVIEWS

• The formative interviews recruited six accessibility QA professionals from a large technology company and asked about testing practices, instructions, and desired replay-tool features.Participants had at least three years of professional experience in accessibility and QA testing of iOS apps.Source: §3 · p. 3

3.1 Challenges & Benefits of Manual Testing

• Participants valued experiencing accessibility features as end users might, including testers who themselves use those features.A VoiceOver user described this perspective as helping them test the feature as intended.Source: §3.1 · p. 4
• Participants from three teams described an overwhelming scope of features and scenarios that led them to target only a few key tests.They also described repetitive work and concern about missing app views or elements.Source: §3.1 · p. 4
• Participants said manual testing takes time to write and maintain, instructions can become outdated, and automated tests can break or miss scenarios.Four participants also mentioned difficulty finding time for testing amid frequent app updates and product milestones.Source: §3.1 · p. 4

3.2 Testing Process

• Participants tested accessibility throughout product lifecycles, including annually as features were added and on regular interface release cycles.Their work included manual feature testing and, for some, writing accessibility frameworks and automation code.Source: §3.2 · p. 4
• Visual-feature tests typically toggle a setting and check UI rendering, while accessibility-service tests enable the service and navigate tasks or screens.VoiceOver is given as an example of an accessibility service.Source: §3.2 · p. 4

3.3 Granularity and Availability of Manual Accessibility Test Instructions

• Manual test instructions were common in databases and bug-tracking tools, and some participants made them as repro steps for engineers less familiar with accessibility features.Participants aimed to make such instructions specific when the feature under test might be unfamiliar to engineers.Source: §3.3 · p. 4
• One team’s annual regression suite covered 10 apps and 300 cases, with instructions ranging from concrete steps to abstract goals that assumed tester expertise.Test cases typically named the platform, feature, and app, but only some specified steps or expected results.Source: §3.3 · p. 4

3.4.1 F0: Natural Language Interpretation and Replay.

• AXNav aims to interpret existing manual instructions at multiple levels of abstraction and replay them without requiring testers to write and update fully automated tests.Participants already had large databases of manual testing instructions and wanted more workflow automation.Source: §3.4.1 · p. 4

3.4.2 F1: Quickly Navigate and Visualize Executed Steps.

• Chapter labels and an action cursor help testers revisit video segments to inspect system actions and screens for potential bugs.Participants requested repeated review of portions of the video.Source: §3.4.2 · p. 4
• Two participants requested replay on a live local device with opportunities to take control, but AXNav did not support this because of architecture constraints.A VoiceOver user was unable to interact with the UI in the video format.Source: §3.4.2 · pp. 4–5

3.4.3 F2: Flag Potential Issues.

• AXNav adds heuristic-based chapter labels at potential issue locations, responding to requests to flag visual and navigation problems.Requested examples included text resizing, wrong navigation order, missing labels, and elements unavailable for navigation.Source: §3.4.3 · p. 5
• Some participants wanted screenshots for bug reports, which could also support postprocessing to flag visual accessibility issues.They described screenshot capture and clipping as time-consuming.Source: §3.4.3 · p. 5

3.4.4 F3: Realistic VoiceOver Navigation and Captioning.

• Participants wanted VoiceOver replay to resemble how users navigate, rather than activating each UI element directly as a sighted user might.Three participants also identified the VoiceOver captions panel as important to include.Source: §3.4.4 · p. 5

3.4.5 F4: Perform Tests With and Without Accessibility Features.

• AXNav repeats most tests with the accessibility feature on and then off so testers can check whether the system returns to the correct state.Participants’ scripts commonly included both conditions.Source: §3.4.5 · p. 5

• AXNav interprets natural-language accessibility tests, replays them on a mobile device, and produces a chaptered, navigable video annotated with heuristics that flag potential issues.Source: §4 · p. 5
• The system supports VoiceOver, Dynamic Type, Bold Text, and Button Shapes, selected to cover testing needs across different modalities.The authors note that support could be extended to other accessibility and device features.Source: §4 · p. 5
• AXNav provisions and stages a cloud iOS device, navigates the app to execute tests, and collects and processes the results.Source: §4 · p. 5

4.1 Device Allocation and Control

• AXNav provisions a remote cloud iOS device, installs the target app, selects the accessibility feature, and prepares the device before launching the app and recording.A custom app silences system notifications, controls recording, and interacts with assistive technologies; for VoiceOver, it also enables the caption panel and sets the speaking rate to 0.25.Source: §4.1 · p. 5

4.1.1 Accessibility Feature Control and Replay.

• For Dynamic Type tests, AXNav captures the target screen at each of four increased text sizes and once with Dynamic Type off, enabling size-by-size comparison.The system relaunches the app and navigates to the target screen for each capture.Source: §4.1.1 · p. 5
• For Bold Text and Button Shapes, AXNav navigates to the target screen with and without the feature enabled and saves paired screenshots; VoiceOver instructions are replayed with VoiceOver on and off.Source: §4.1.1 · pp. 5–6

4.2 Test Planning and Execution

• Three LLM agents—planner, action, and evaluation—translate test instructions into steps, execute them on a live device, and use results to improve navigation plans.Pixel-based models recognize UI elements, text, and icons; device tools send touch, keyboard, and VoiceOver inputs.Source: §4.2 · p. 6

4.2.1 Test Plan Proposal.

• The planner proposes a tentative step list from the test instructions, app name, and detected screen elements, with steps specifying actions, success criteria, and execution status.Its instructions accommodate unexpected states by allowing backward traversal and imperfect plans that can be revised later.Source: §4.2.1 · pp. 6–7, Figure 4 · p. 7

4.2.2 Representing UI Elements to the Agents.

• AXNav represents detected UI elements as an ordered plain-text list with IDs, element types, text, and bounding-box coordinates, using pixel-based recognition that is agnostic to UI framework.Keyboard detection filters keyboard elements from the list except submit buttons.Source: §4.2.2 · p. 7

4.2.3 Mapping from Plan Steps to Concrete Actions.

• For each plan step, the action agent maps natural-language instructions to a UI element, checks feasibility, and selects a device action such as tap, swipe, text entry, or stop.The agent returns a function call with relevant UI IDs and elements.Source: §4.2.3 · p. 7

4.2.4 Executing VoiceOver Actions.

• VoiceOver execution uses gestures for reading elements, activating targets, and scrolling; activation navigates by swiping to the target before double-tapping, matching a pattern preferred by formative-study participants.Read-all traversal is capped at 50 elements to limit time and avoid loops or infinite-scroll screens.Source: §4.2.4 · pp. 7–8

4.2.5 Evaluation and Replanning.

• The evaluation agent judges each action from the goal, plan, action, and before-and-after UI detections, then returns success, failure, or task completion with an explanation for replanning.A negative evaluation prompts the planner to revise the plan from the current step onward.Source: §4.2.5 · p. 8

4.3 Test Heuristics

• AXNav flags four potential issue types: VoiceOver navigation loops, missing VoiceOver elements, Dynamic Type resizing failures, and Button Shapes failures.Figure 5 illustrates examples of Button Shapes and Dynamic Type flags.Source: §4.3 · p. 8

4.3.1 VoiceOver loop detection and missing VoiceOver elements.

• During VoiceOver read-all and activation commands, AXNav flags a loop when an element is revisited and attempts to resume navigation below the looped element.Source: §4.3.1 · p. 8

4.3.2 Missing VoiceOver elements.

• AXNav flags a missing VoiceOver element when its UI detector identifies an element that cannot be found during activate-from-coordinates navigation.Source: §4.3.2 · p. 9

4.3.3 Dynamic Type.

• The Dynamic Type heuristic compares paired screenshots one size increment apart, matches text with OCR and fuzzy string similarity, and flags text whose bounding box grows by less than 10%.It also matches adjacent icons to text and applies the same 10% area-growth threshold; unmatched text elements are excluded.Source: §4.3.3 · p. 9

4.3.4 Button Shapes.

• The Button Shapes heuristic checks a screenshot with the feature enabled, flagging underlined text inside clickable containers and un-underlined text predicted as clickable outside them.Underline detection uses image processing to find a horizontal line spanning at least 75% of a text patch’s width.Source: §4.3.4 · p. 9

4.4 Output Video Generation

• AXNav records device screens, action timestamps, actions, and activated UI elements, then adds chapter markers for test steps and heuristic-flagged issues.Chapters are named; video overlays also mark actions and potential accessibility issues, and exported videos are sped up 2.5×.Source: §4.4 · p. 9
• Tables 1 and 2 organize navigation-replay results by accessibility feature and Easy or Hard difficulty, reporting full success, partial success, failure, and overall accuracy.Source: Table 1 · p. 9, Table 2 · p. 9
• Video overlays label taps with crosshairs, scrolling with arrows, and potential accessibility issues with orange or cyan bounding boxes.Exported videos are sped up by a factor of 2.5 to minimize pauses from LLM-agent latency.Source: §4.4 · p. 9

5.1 Regression Testing Dataset

• The regression dataset contains 62 manual test cases from five apps, covering VoiceOver, Dynamic Type, Button Shapes, and Bold Text.The suite was not designed for automation, and its high-level instructions assume expert knowledge; the authors chose it as a representative set of real-world accessibility tests.Source: §5.1 · pp. 9–10

5.2 Free Apps Dataset

• The free-app dataset contains 20 tests across five apps and four supported accessibility features, with tests drafted by a researcher and reviewed by an expert QA tester.The apps were randomly selected from a list of the 100 most popular free App Store apps.Source: §5.2 · p. 10

5.3 Accuracy Results

• Tests were rated Easy or Hard, replayed from specified initial states, and manually scored by two researchers for navigation success, partial success, and accessibility-test success.Two authors independently rated difficulty and reconciled differences; two researchers independently scored test outcomes and reconciled differences.Source: §5.3 · p. 10
• 85.5% overall replay success on regression tests (95.5% easy; 61.1% hard), alongside 78% accessibility-test success.Source: §5.3 · p. 10
• 70.0% overall replay success on free-app tests (83.3% easy; 64.3% hard), alongside 15.0% accessibility-test success.Source: §5.3 · p. 10
• Replay failures included selecting the wrong collection item, missing app functionality, stopping short of offscreen controls, and planner loops.The authors identify these as areas for future improvement.Source: §5.3 · p. 10

6.1 Procedure

• The study used 10 one-to-one sessions with professional accessibility testers, showing three AXNav videos covering VoiceOver, Dynamic Type, and Button Shapes.The videos included deliberately imperfect navigation and some false-positive flags to conservatively demonstrate system performance.Source: §6 · p. 10, §6.1 · pp. 10–11

6.2 Participants

• Participants were full-time accessibility QA testers or engineers at one large technology company; all were sighted and did not use screen readers.Researchers analyzed interview transcripts through thematic analysis and reported descriptive statistics for Likert ratings.Source: §6.2 · p. 11, §6.3 · p. 11

6.4.1 Performance of the Automatic Test Navigation.

• VO navigation similarity averaged 4.60 (SD = 0.52, N = 10), BS similarity averaged 4.35 (SD = 0.75, N = 10), and DT similarity averaged 3.39 (SD = 0.78, N = 9).Participants noted that AXNav relaunches the app between text-size tests, unlike human testers who often adjust text size in-app; some saw this as an additional test layer.Source: §6.4.1 · pp. 11–12
• Participants requested more exploratory and alternative navigation, deeper UI-structure testing, and scrolling during display-feature tests.Requests included exploring nested content, non-linear VoiceOver routes, and both swiping and tapping gestures.Source: §6.4.1 · p. 12
• Only 2 of 10 participants noticed the VO video’s error of sharing a show instead of an episode; participants described possible over-reliance on automatic navigation.One participant suggested making the instruction more specific to help the agent find the intended episode.Source: §6.4.1 · p. 12

6.4.2 Identifying Accessibility Issues with Automatic Navigation.

• All participants found at least one significant issue in the VO, BS, or DT videos, and heuristic usefulness ratings ranged from 3.67 to 4.75 across the three cases.Participants said chapter flags helped direct attention to subtle potential issues, though the heuristics sometimes produced false positives.Source: §6.4.2 · p. 12
• False negatives may discourage testers from inspecting unmarked areas, while inexperienced testers may mistake false positives for real bugs.Participants suggested explaining how heuristics were generated and applied to reduce over-reliance.Source: §6.4.2 · p. 12

6.4.3 Integration in Accessibility Testing Workflow.

• Participants rated AXNav’s usefulness in their existing workflow 4.70 (SD = 0.48, N = 10) if it performed extremely well, and 3.95 (SD = 0.96, N = 10) in its current form.Participants envisioned automated test planning, background test runs, and using generated videos to validate manual tests.Source: §6.4.3 · pp. 12–13

7 DISCUSSION

• The authors frame AXNav as a complement to manual testing, which remains important, and report that testers found it useful in their workflows.They also present it as an initial exploration of LLM-based task automation for accessibility testing.Source: §7 · p. 13

7.1 Differences between automated navigation and manual testing

• AXNav's VoiceOver workflow uses forward swipes to find a target, which may differ from users who know the app and navigate directly.The authors suggest exploring alternative interaction patterns.Source: §7.1 · p. 13

7.2 Improving navigation performance

• Replay errors can arise when AXNav lacks app knowledge or cannot determine when to stop.The authors propose fine-tuning on successful paths, adding app knowledge to prompts, or considering multimodal models.Source: §7.2 · p. 13

7.3 Mitigating errors and over-reliance

• AXNav may produce errors, and only 2 of 10 participants spotted the VoiceOver navigation error, raising a risk of over-reliance.The authors suggest evaluating navigation paths and adding transparency signals such as confidence scores or explanations.Source: §7.3 · pp. 13–14

7.4 Limitations in the User study

• The user study showed all participants the same three mostly successful replay videos, and they did not create tests or generate videos themselves.The authors propose a longitudinal study to examine how testers instruct the system and interact with its output.Source: §7.4 · p. 14

7.5 Accessibility of AXNav

• AXNav's video output is not accessible by default to screen reader users, and visual annotations need descriptions for them to use it.A formative-study screen reader user requested live replay on a local device to retain control.Source: §7.5 · p. 14

7.6 Accessibility feature support and generalizability

• AXNav supports four accessibility features and only surfaces a subset of potential issues; future work could add features, inspection tools, and issue summaries.The supported features can be extended where operating-system APIs allow control; the system currently works only on iOS, though its architecture and workflow should extend to other platforms with suitable APIs.Source: §7.6 · p. 14

7.7 More applications of the AXNav system

• The authors identify possible uses of AXNav for helping novice developers learn accessibility-feature behavior and for reconstructing other natural-language UI tests.Examples include manual UI tests, bug reports, and reproduction steps.Source: §7.7 · p. 14

8 CONCLUSION

• AXNav supports interpretation and replay of natural-language accessibility tests, with 70% and 85% navigation replay success and a 10-tester study reporting workflow usefulness.The study involved professional accessibility testers.Source: §8 · p. 14