Source-linked AI summary
Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok
TL;DR
The paper asks how reliably computer-use agents support blind screen-reader users in everyday desktop workflows, where prior evaluations provide limited real-world evidence. It conducts a three-week diary study with OLLA and re-executes participant commands across models, finding that GPT-5 had the highest observed success rate at 52.5% while users also sought collaborative support beyond automation.
Problem
Existing evidence is limited on how reliably CUAs support blind screen-reader users in everyday desktop workflows and where nonvisual execution breaks down.
Method
A three-week diary study with 8 blind users used OLLA to collect 1,258 commands and execution traces, followed by controlled re-execution with four additional models.
Results
GPT-5 achieved the highest observed success rate at 52.5%, while trace analysis identified recurring grounding, planning, constraint-tracking, and termination failures.
Takeaways & Limitations
Blind users envisioned CUAs as collaborative support for understanding interfaces, recovering from breakdowns, troubleshooting, learning workflows, and improving efficiency beyond end-to-end automation.
Takeaways & Limitations
The findings primarily reflect blind screen-reader users and desktop applications, excluding low-vision users, other visual-impairment conditions, and web applications.
Abstract
from arXiv · showhide
Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.
1 Introduction
This paper examines whether computer-use agents can support blind screen-reader users in real-world desktop workflows, where prior evidence is limited. It introduces OLLA and evaluates CUA effectiveness, breakdowns, and uses beyond automation.
- Motivation: Blind users face persistent GUI barriers because screen readers provide sequential, keyboard-driven interaction with interfaces designed around spatial layouts and point-and-click use.Reported difficulties include locating controls, navigating nested structures, understanding dynamic updates, recovering from errors, and adapting to interface changes.
- Research gap: Prior accessibility studies largely used simulated evaluations, persona prompting, or narrow laboratory tasks, leaving real-world CUA performance for blind users unclear.The paper also asks where nonvisual execution breaks down and how users envision support beyond end-to-end automation.
- Approach: The study used OLLA, a screen-reader-accessible layer that lets blind users issue commands, monitor execution, and review agent actions without modifying the underlying CUA.OLLA was deployed with GPT-5 during authentic workflows spanning 12 applications, while recording screenshots, UI trees, model responses, generated actions, and interaction histories.
- Findings: 52.5% was GPT-5’s highest success rate among evaluated models, while trace analysis identified grounding, planning, constraint-tracking, and termination failures.The paper reports these findings as evidence that current CUAs remain unreliable despite completing some nonvisual tasks.
- Beyond automation: Interviews showed that blind users envision CUAs as collaborative support for interface learning, troubleshooting, recovery from unfamiliar states, and repetitive or technical workflows.This broadens the target from autonomous task completion to assistance during interaction and breakdowns.
- Contributions: The three-week deployment collected 1,258 blind user-issued desktop commands, paired with detailed execution traces for human-centered evaluation.The dataset supports analysis of real-world nonvisual workflows and systematic agent behavior.
2 Background
Background work spans broad CUA benchmarks and accessibility research, but existing evaluations do not fully represent blind users’ everyday nonvisual desktop work. Accessibility also does not guarantee usable interaction.
- CUA benchmarks: CUA benchmarks evaluate language-guided interaction, action prediction, visual grounding, and task completion across web, desktop, operating-system, and mobile environments.Examples include WebShop, Mind2Web, WebArena, OSWorld, Windows Agent Arena, Android in the Wild, and AndroidWorld.
- Expanded evaluation: Newer benchmarks examine online realism, workplace autonomy, safety, policy compliance, and accessibility-related goals.These include Online-Mind2Web, TheAgentCompany, ST-WebAgentBench, and BLIND-ACT.
- Open gap: Existing benchmarks do not fully capture how CUAs support blind users in everyday computer tasks, where they fail, or how assistive agents should be designed.The paper positions its human-centered study as addressing this gap.
- Accessibility barriers: Blind users typically rely on screen readers, but technically accessible controls may remain difficult to locate, understand, and operate.The mismatch between visually organized GUIs and sequential screen-reader interaction is amplified by application heterogeneity, complex shortcuts, and shifting interaction patterns.
- Prior solutions: Prior systems reduce interaction burdens through interface adaptation, structured navigation, context-aware guidance, uniform mechanisms, natural-language commands, and mixed-initiative interaction.These directions motivate assistive CUA designs beyond purely autonomous operation.
3 Evaluating CUAs as Assistive Agents for Blind Users
The paper evaluates CUAs as assistive agents through a naturalistic diary study with blind screen-reader users and controlled re-execution of their commands. It combines interaction traces, annotation, and interviews to examine effectiveness and breakdowns.
- Study design: An IRB-approved three-week diary study collected everyday nonvisual computer-use data from 8 blind screen-reader users.Diary studies were selected to capture repeated technology experiences in naturalistic settings over time.
- Accessible interface: OLLA provides a screen-reader-accessible interaction layer for issuing commands, monitoring execution, and reviewing agent actions nonvisually.The layer operates over existing CUAs without changing their underlying architecture or reasoning process.
- Cross-model evaluation: The same participant-issued commands were re-executed independently across four additional CUA models in fresh live application instances.Resetting each model-command pair prevented reconstructed traces or cross-execution state from affecting comparisons.
- Annotation: Four annotators independently established minimum reference steps and labeled execution outcomes as success, partial completion, or failure.They also annotated applicable failure categories using a shared protocol.
- Analysis: Trace analysis reviewed commands, model responses, selected actions, UI-tree states, screenshots, and interaction histories to locate execution breakdowns.Post-study interviews were analyzed with hybrid reflexive thematic analysis to interpret helpful, confusing, incomplete, and difficult-to-monitor behavior.
4 RQ1. Effectiveness of CUAs
The evaluation combines real-world use with controlled cross-model re-execution of 1,258 commands across 12 applications. CUAs succeeded most often on direct, visible operations but remained unreliable on tasks requiring discovery, multiple steps, or constraint maintenance.
- Study agent and baselines: The evaluation compared GPT-5, Claude Sonnet, Gemini 2.5 CU, UI-TARS, and Qwen3-VL using the same participant-issued commands and interface context.Models received the task command, UI tree, screenshot, and recent interaction history, then generated structured actions.
- Study agent and baselines: 1,258 commands across 12 desktop applications were normalized into 304 task intents and analyzed as individual task attempts.Commands with different wording or parameter values were grouped when they represented the same underlying task.
- Task outcomes and step progress: 52.5% was GPT-5’s observed success rate, followed by Claude Sonnet at 48.5%, Gemini 2.5 CU at 43.9%, UI-TARS at 39.8%, and Qwen3-VL at 37.9%.GPT-5’s 4.0-percentage-point advantage over Claude Sonnet was not statistically significant in a paired McNemar test (χ2 = 3.25, p = .071).
- Task outcomes and step progress: Partial completion ranged from 33.3% to 34.9% across models, with GPT-5 completing 68.3% of required steps on average before breakdown.StepProgress_i = c_i/r_i, where c_i is completed steps and r_i is required reference steps.
- Interactions handled more successfully: Successful interactions concentrated on direct property changes, single-step operations, navigation, information retrieval, and explicitly parameterized editing tasks.These tasks had direct mappings between commands and readily identifiable controls.
- Interactions handled more successfully: Multi-step reasoning, hidden-control discovery, and multiple-constraint maintenance were less consistently successful, with performance also varying by application structure, control visibility, and task type.The reported patterns characterize observed conditions rather than reliable performance across all task instances or interface states.
- Application-level variation: The success rates are conditional on tasks participants chose to attempt because some avoided sensitive tasks or stopped attempting task types after repeated failures.They therefore do not capture tasks participants considered but elected not to delegate.
5 RQ2. Breakdowns During Nonvisual Task Execution
CUAs failed during nonvisual execution not only because they misread interfaces, but also because they struggled to discover hidden paths, preserve intent and context, and recognize completion.
- UI Grounding and Hidden Path Discovery: 24.6% of GPT-5 failures involved ungrounded action generation, including hallucinated controls, fabricated control types, or misaligned coordinates.Across all models, grounding errors accounted for 22.6% of failures; ribbon-based applications were especially challenging.
- UI Grounding and Hidden Path Discovery: 21.4% of GPT-5 failures involved hidden paths, where agents handled visible options but failed to navigate dialogs, menus, or deeper variants.Custom margins and 15-by-15 tables required intermediate navigation beyond exposed controls.
- Prior Knowledge Overriding UI-Aware Reasoning: 17.1% of GPT-5 failures reflected learned procedures, while 11.4% reflected default collapse that ignored user-specified options.Agents sometimes transferred Word-like procedures to OneNote or selected Normal or Narrow margins instead of opening the custom dialog.
- Prior Knowledge Overriding UI-Aware Reasoning: 5.9% of GPT-5 failures involved structured intent, such as preserving exact spreadsheet targets, functions, ranges, operators, grouping, or ordering.These failures arose during intermediate representation formation rather than control location.
- State Maintenance Across Task Steps: 20.6% of GPT-5 partial-completion cases lost one or more user constraints after initially correct actions.Examples included changing a font family without its requested size or inserting a page number without centered alignment.
- State Maintenance Across Task Steps: 17.6% of GPT-5 partial-completion cases involved premature stopping, pointless continuation, or repeated actions instead of recognizing that execution was no longer progressing.Other state failures included losing prior context, coordinating across applications, and missing final commit actions.
6 RQ3. CUAs Beyond Automation
Participants envisioned CUAs as collaborative aids that explain interfaces, provide situated help, preserve user control, and support learning rather than simply automate complete tasks.
- Understanding Interfaces: Participants wanted CUAs to explain charts, layouts, controls, menus, dialogs, settings, and document structure before execution.This support would help users understand unfamiliar or visually organized interfaces and decide what to do next.
- Situated Assistance: Participants wanted targeted assistance when stuck, including locating hidden controls, interpreting dialogs, identifying changes, troubleshooting states, and choosing next steps.They did not necessarily want agents to take over the entire task.
- User-Controlled Execution: Participants wanted agents to explain planned actions, ask for missing information, and confirm assumptions before consequential or difficult-to-verify actions.They were especially cautious about settings, files, deletion, and sensitive personal or financial information.
- Learning and Productivity: Participants saw CUAs as tools for learning application structure, receiving screen-reader-relevant guidance, practicing workflows, and handling tedious or technical work.The desired outcome was to help users perform workflows independently in the future.
7 Discussion and Future Work
The discussion argues for CUA designs that support clarification, recovery, and user-centered assistance beyond end-to-end automation, while using the dataset to study and improve failures.
- Dataset and Future Evaluation: The dataset captures commands, UI states, screenshots, model responses, generated actions, and interaction histories from blind users’ nonvisual workflows.Future annotations could include uncertainty, clarification opportunities, rejected actions, recovery attempts, and decision changes.
- Improving Agent Behavior: Failure analysis identifies reward-guided improvement opportunities for wrong paths, repeated actions, and continued execution despite little progress.The discussion connects supervision to both final outcomes and intermediate judgments about useful, safe, or correct behavior.
- Beyond Automation: Participants’ reflections support mixed-initiative CUAs that provide targeted help with unfamiliar interfaces, troubleshooting, and learning rather than only full automation.Potential directions include interface adaptation, context-aware guidance, and natural-language control.
8 Conclusion
The paper evaluates CUAs in everyday desktop workflows for blind screen-reader users and finds meaningful task completion alongside recurring execution failures and demand for collaborative support.
- Conclusion: The study combined a three-week diary study with cross-model evaluation of 1,258 participant-issued commands in real-world nonvisual desktop contexts.The evaluation examined grounding, hidden-path discovery, state maintenance, constraint preservation, and termination recognition.
- Conclusion: Participants wanted CUAs as collaborative support for understanding interfaces, recovering from breakdowns, and learning workflows rather than autonomous replacements.This conclusion extends the paper’s focus beyond end-to-end task automation.
Limitations
The study’s conclusions are bounded by its participant sample, Windows desktop prototype, English-only setting, desktop-application focus, and selection of five models rather than all available systems.
- Participant Pool: The participant pool included blind screen-reader users but excluded low-vision users, other visual-impairment groups, and people under 18.Findings therefore primarily reflect screen-reader-mediated desktop use and may not generalize to different access needs or age groups.
- Prototype and Operating System Scope: The prototype was implemented for Windows with Microsoft UI Automation, limiting direct generalization to operating systems with different accessibility infrastructures.Other platforms may expose different accessibility APIs, permission models, screen-reader behavior, or UI-tree completeness.
- Language Scope: The study used English commands, feedback, surveys, and interviews, leaving multilingual and localized screen-reader workflows unexamined.Language mixing, localized labels, and differing shortcut conventions may shape agent performance and user expectations.
- Application Scope: The evaluation focused on desktop applications and did not include dynamic web applications.Web interfaces can involve JavaScript-rendered content and divergence among DOM structure, accessibility-tree information, and visual state.
- Model Scope: The study evaluated five agent-capable models rather than every available model, so its results characterize current capabilities rather than provide a definitive ranking.Each model required re-executing 1,258 commands with logged traces and manual outcome verification, while rapidly evolving models may show different strengths or failures.
Ethical Considerations
The study was designed for accessible participation by blind screen-reader users, with informed consent, privacy protections, longitudinal use in participants’ environments, and interviews focused on user experience and expectations.
- Accessible Participation: Recruitment, consent, installation, instructions, surveys, and interviews were designed to be accessible with screen readers.Participants were informed about study procedures and could stop participation or skip tasks.
- Recruitment and Eligibility: Participants were recruited from a previously consented accessibility-study contact list and through snowball sampling using preferred communication methods.Eligibility required self-identifying as blind, regular screen-reader use, desktop-application experience, and access to a personal computer.
- Study Conduct: The study involved 8 blind screen-reader users who used OLLA across 12 desktop applications during three weeks in their own environments.Participants were encouraged to issue at least 10 commands daily and report both successful and unsuccessful attempts.
- Consent and Privacy: Participants provided informed consent, could withdraw at any time, and had personally identifying information removed before analysis.The protocol recognized that screenshots, UI trees, commands, model outputs, and action traces could contain sensitive personal-computer data.
- Interviews and System Context: Post-study interviews used interaction logs and surveys to examine breakdowns and participants’ expectations for guidance, troubleshooting, learning, control, privacy, and trust.OLLA was developed as a screen-reader-friendly accessibility layer because existing CUAs were difficult to deploy directly with blind participants.
B.3 Statistical Analysis
The analysis quantified uncertainty in model success rates with Wilson confidence intervals and compared paired command outcomes; the observed GPT-5 advantage over Claude Sonnet was not statistically significant.
- Statistical Procedures: 95% Wilson confidence intervals and paired command-level comparisons were used to quantify uncertainty in observed success rates.The evaluation reports these procedures for cross-model performance analysis.
- Model Comparison: 52.5% was GPT-5’s success rate, compared with 48.5% for Claude Sonnet.The corresponding 95% confidence intervals were 49.8–55.3% and 45.7–51.3%, respectively.
- Model Comparison: The 4.0-percentage-point GPT-5 advantage was not statistically significant in a paired McNemar test (χ2 = 3.25, p = .071).The authors caution against interpreting the highest observed rate as a definitive performance advantage.
- Analysis Context: The evaluation artifacts include eight command-derived task categories and fixed model configurations, with all models limited to 2,048 output tokens.Table 3 covers category distributions and operational definitions; Table 4 covers models and execution configurations.
C Task Outcomes by Model and Application
Across 1,258 participant-issued commands, performance varied by model and task demands: GPT-5 had the highest observed success rate, while multi-step reasoning, hidden-control discovery, and constraint maintenance were less consistent.
- Outcomes by Application: Task-level outcomes were evaluated across models on participant-issued commands (N = 1,258), with application-level counts distinguishing success, partial completion, and failure.Application-level step-progress analysis reports average fractions of reference steps completed before breakdown for partial completions.
- Breakdown Analysis: Six trace-analysis breakdown patterns covered interface grounding, context maintenance, prior-knowledge reliance, default selection, multi-step planning, and constraint binding.Figures 5–7 visualize these failures across interface grounding, situated reasoning, and multi-step execution.