Source-linked AI summary
MCP-Driven Accessibility Tree Standardization for AI-Powered Screen Reader Agents
Vishnu Ramineni, Nitin Saksena, Akash Kumar Agarwal, Darshan Mohan Bidkar, Balakrishna Pothineni, Durgaraman Maruthavanan, Lokesh Butra, Siva Kumar Chintham
TL;DR
LLM screen-reader agents lack a unified way to access semantically rich interface state across heterogeneous accessibility APIs, while screenshot and tree-based perception involve different trade-offs. The paper proposes an MCP accessibility server with normalized tree resources, navigation tools, and persistent disability profiles, and its qualitative analysis suggests lower integration and context-token costs without resolving native accessibility latency. The work remains an architectural proposal rather than an empirical cross-platform evaluation.
Problem
Accessibility support lacks a shared platform-independent schema, leaves screenshot-versus-tree trade-offs insufficiently examined through standardized mediation, and generally does not persist user disability profiles across sessions.
Method
The paper specifies an MCP server with platform adapters, an ARIA-aligned normalization layer, scoped accessibility-tree and profile resources, and navigation tools, then compares it with prior perception strategies.
Results
The qualitative analysis suggests MCP mediation can reduce platform-specific integration complexity and context-token consumption while not resolving latency inherited from native accessibility APIs.
Takeaways & Limitations
A standardized MCP accessibility layer provides a single protocol surface for agent developers while preserving structured accessibility semantics and supporting persistent accessibility profiles.
Takeaways & Limitations
The evaluation is architectural and comparative: no working cross-platform prototype was built, and Table I and Fig. 2 contain qualitative literature-grounded judgments rather than common-benchmark measurements.
Abstract
from arXiv · showhide
Large language model (LLM) agents that interact with graphical user interfaces increasingly rely on either raw screenshots or platform-specific accessibility application programming interfaces (APIs) to perceive interface state. Both approaches have limitations for assistive applications: screenshot-based perception lacks the semantic roles and relationships required by screen readers, while platform-specific APIs such as Windows UI Automation, macOS Accessibility, Android AccessibilityService, and web ARIA require separate integrations for each platform. This paper proposes an architecture that uses the Model Context Protocol (MCP) as a unified transport and schema layer between heterogeneous accessibility frameworks and LLM-based assistive agents. An MCP accessibility server exposes ARIA-aligned roles, labels, states, and focusable-element hierarchies through a platform-independent representation, enabling consistent interaction across operating systems and applications. The framework also introduces an MCP resource model for persisting user accessibility preferences across sessions. The architecture is analyzed with respect to three research questions: protocol extensibility for accessibility-tree representation, latency and semantic fidelity trade-offs between accessibility trees and screenshot-based perception, and support for persistent accessibility profiles through MCP resources. Rather than presenting an empirical implementation, this work contributes a conceptual framework supported by comparative analysis of accessibility APIs, GUI agent architectures, and the MCP specification. The analysis suggests that a standardized MCP accessibility layer can reduce platform-specific integration complexity while preserving the semantic information required for accessible AI agents, providing a foundation for future implementation and evaluation.
I. INTRODUCTION
Screen readers traditionally consume platform-specific semantic accessibility trees, while LLM GUI agents commonly use screenshots or serialized trees. The paper proposes an MCP server layer to standardize accessibility exposure across platforms.
- Screen readers derive roles, names, and states from operating-system and browser accessibility layers.
- LLM GUI agents primarily perceive interfaces through screenshots or textual accessibility-tree serializations.
- MCP provides a standardized JSON-RPC interface for discovering and invoking external resources and tools.
- The proposed MCP server sits between native accessibility APIs and MCP-capable agents, exposing interface state as resources and actions as tools.
A. Problem Statement
The paper identifies missing standardization, unresolved perception trade-offs, and stateless handling of user accessibility preferences as three deficiencies motivating its architecture. It contributes an MCP design, qualitative analysis, persistent profiles, and an implementation roadmap.
- No shared platform-independent schema lets accessibility subsystems expose trees to LLM agents, forcing per-platform extraction logic.
- Screenshot and tree perception trade-offs remain qualitatively discussed rather than examined through a standardized protocol mediating both modalities.
- Existing assistive agents largely re-derive disability-profile preferences each session instead of persisting them across invocations.
- The paper contributes an MCP accessibility-server architecture, literature-grounded trade-off analysis, persistent disability profiles, and steps toward validation.
II. LITERATURE REVIEW
Prior GUI-agent research contrasts vision, text, and hybrid perception, while accessibility work remains fragmented across specialized systems. The review positions standardized MCP mediation as a response to platform variation, extraction latency, and token limits.
- GUI-agent research distinguishes pure-vision, pure-text, and hybrid approaches by input modality.
- Accessibility-tree representations preserve semantic labels and use fewer tokens, but extraction can be slow and platform-dependent.
- Tree extraction latency compounds across long-horizon tasks, while returned trees vary in completeness across platforms and applications.
- Tree-search agents serialize accessibility trees under fixed token budgets, requiring bounded transmission or summarization for context-limited models.
- ARIA supplies a canonical role, state, and property vocabulary that can ground the proposed MCP resource schema.
- Adjacent AI-accessibility systems address voice interaction, ADA detection, sign language, or chatbot accessibility rather than standardized agent-side perception.
III. METHODOLOGY
The study uses design-science analysis rather than model training or controlled user research. It evaluates the architecture through MCP specification alignment, semantic mapping, and comparison with reported prior-system characteristics.
- The study evaluates the proposal through formal architecture specification against MCP primitives.
- It structurally maps ARIA and platform accessibility semantics onto the MCP primitive set.
- Comparative analysis uses perception strategies documented in the reviewed literature.
- The approach fits questions about protocol extensibility and architectural feasibility, which the paper treats as outside a benchmark requiring a working multi-platform implementation.
A. Design Constraints
The architecture is shaped by backward compatibility, ARIA-aligned schema reuse, bounded context consumption, and falsifiable evaluation criteria.
- Backward compatibility requires constructing the server from data exposed by existing platform accessibility APIs.
- Schema reuse maps platform roles and states onto ARIA vocabulary wherever compatible mappings exist.
- The proposal specifies navigation, latency, and semantic metrics so future implementation can be evaluated rather than remaining purely descriptive.
A. Architectural Overview
The architecture places an MCP Accessibility Server between native accessibility subsystems and an MCP-capable agent, using adapters, normalization, resources, tools, and updates.
- Platform adapters translate web, Windows, macOS, and Android accessibility trees into a common internal representation.
- The MCP interface exposes the normalized tree as resources and navigation actions as tools between the server and agent client.
- A normalization layer maps platform-specific roles and states to a shared ARIA-aligned vocabulary, with generic roles and platform hints for unmatched cases.
- Resource-update notifications report interface mutations so the agent does not need to poll.
- Scoped tree resources provide bounded subtrees, while focus, activate, read, and query tools support interaction and context control.
C. Disability Profile Persistence
The proposed profile resource persists user accessibility preferences across sessions, while the comparative analysis remains qualitative and identifies integration and token-cost advantages without eliminating native latency.
- Disability Profile Persistence: The profile resource stores preferences such as reading granularity, verbosity, motor-accessibility thresholds, and destructive-action confirmation behavior.
- Disability Profile Persistence: Addressable profiles can be retrieved across devices and application restarts without requiring the agent to implement storage.
- Comparative Analysis: The comparison assesses semantic fidelity, integration cost, per-turn latency, and context-token cost on a Low/Medium/High ordinal scale.
- Comparative Analysis: Scoped queries reduce context-token cost relative to whole-tree serialization, while the MCP layer inherits native accessibility-API latency.
- Comparative Analysis: Figure 2 visualizes the same ordinal assessments on a 1–3 scale, which represents qualitative ratings rather than measured quantities.
VI. DISCUSSION
The paper frames accessibility support for AI agents as protocol standardization rather than per-agent scraping. MCP mediation reduces integration and token costs through normalization and scoped queries, but inherits native accessibility-API latency.
- Treating the accessibility tree as a resource and navigation actions as tools lets normalization absorb platform translation work from clients.
- The architecture partially addresses latency and fidelity trade-offs because scoped queries reduce token cost while native API latency remains inherited.
- The proposal provides infrastructure beneath assistive features such as voice or sign-language layers, which still require structured interface state.
A. Limitations
The evaluation is qualitative and architectural because no cross-platform implementation or common benchmark was built. Practical deployment also faces incomplete role mappings and persistent accessibility-tree data-quality problems.
- No working cross-platform prototype was built, so Table I and Fig. 2 provide qualitative literature-grounded judgments rather than common-benchmark measurements.The unmeasured dimensions include semantic fidelity, integration cost, latency, and related comparative characteristics.
- Fallback handling for platform roles without ARIA equivalents is specified only at a high level and would require a substantial maintained mapping table.The required mapping effort is compared with mapping tables maintained internally by browser engines.
- Native accessibility trees are often incomplete, including unlabeled buttons and images, and this data-quality problem remains independent of the protocol layer.The limitation would persist even if the proposed server were fully implemented.
B. Threats to Adoption
Adoption depends on coordination among assistive-technology vendors and platform accessibility teams, while validation requires implementation, user studies, and mapping-coverage evaluation. The proposed architecture may reduce integration cost and context-token consumption but inherits native accessibility-tree latency.
- Adoption requires assistive-technology vendors and platform accessibility teams to agree on adapters conforming to the proposed schema.The paper characterizes this as a coordination problem rather than a purely technical one.
- The normalization layer must keep pace with changes in platform accessibility APIs over time.
- The proposal may reduce integration cost and context-token consumption without resolving the native latency of accessibility-tree extraction.This latency is inherited from the accessibility layer rather than introduced by the architecture.
- Future validation requires reference adapters, navigation-accuracy studies with screen reader users, and coverage testing of the role-mapping table.These stages would replace qualitative comparison with measured latency, token-cost, completeness, and task-completion evidence.