Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models

1 INTRODUCTION

• Complex GUIs and heterogeneous application and screen-reader shortcuts make basic tasks difficult and require blind users to remember different navigation methods.Prior work reported that blind users were frequently ten times slower than sighted users on similar computer tasks.Source: §1 · pp. 1–2
• Savant uses LLMs to let blind screen-reader users access controls uniformly through natural language, automating action sequences without requiring exact control names.A prior study found that participants considered the absence of uniform access across applications, screen readers, and platforms a significant drawback.Source: §1 · p. 2
• A command such as “set the margin to narrow” can be interpreted as a control action and executed as ordered screen-reader steps, even when phrased differently.The same command can be used across similar applications, avoiding the need to remember distinct shortcuts for each.Source: §1 · p. 2
• SEQ scores were 3 times higher and NASA-TLX scores 3 times lower than with status-quo screen readers; participants accessed controls nearly 4 times faster on average.The study involved 11 blind participants, and command interpretation and automation received an accuracy score of 0.78.Source: §1 · p. 2

2.1 Computer Usability for Blind Users

• Research on blind users’ computer usability remains limited, despite documented issues navigating two-dimensional GUIs with one-dimensional screen-reader navigation.Usability encompasses ease, efficiency, and satisfaction in performing computer tasks.Source: §2.1 · p. 2
• Assistive usability solutions have largely targeted web and mobile applications, while desktop solutions are scarce and often address selected application features.Prior desktop systems include application-specific tools for word processing, spreadsheets, PDFs, email, and Office applications.Source: §2.1 · p. 3
• AccessBolt offers system-wide access to visible controls, but not controls outside the current view, and its sequential navigation can remain time-consuming.Savant is contrasted as shielding users from low-level screen-reader operations and GUI-navigation intricacies.Source: §2.1 · p. 3

2.2 Natural Language Assistance for Blind Users

• Commercial voice assistants provide little support for in-application control tasks, while application-specific assistants are tied to their host applications.Their underlying technologies are closed source, making generalization across applications difficult.Source: §2.2 · pp. 3–4
• Research natural-language assistants have supported web or mobile interactions, but the paper identifies no application-agnostic natural-language assistive technology for consistent desktop interaction.The paper notes that web and mobile prototypes cannot readily be adapted to desktop applications because of differences in underlying software implementations and technologies.Source: §2.2 · p. 4

3.1 Preliminary Data Collection

• The WOZ study with 11 blind participants collected 145 natural-language commands across five applications and annotated each with its corresponding control-value pair.The dataset served as seed data for model training and testing.Source: §3.1 · p. 4

3.2 Overview

• Preprocessing builds an Application Control Tree and Few-Shot Examples Dataset once per application, which runtime reuses to execute user commands.The system has Preprocessing and Runtime components.Source: §3.2 · p. 4

3.3 Preprocessing Component

• The Crawler reveals hidden controls through simulated interactions, then represents controls as tree nodes and access actions as edges.The tree path records the actions needed to reach and adjust a setting.Source: §3.3.1 · p. 5
• Pair Generator enumerates control-value pairs, and Seed LLM generates command-linked triples; researchers filtered the examples to 1,200 across 11 applications.Each FED triple links a natural-language command to a control and value.Source: §3.3.2 · p. 6

3.4 Run-Time Component

• At runtime, BERT selects semantically relevant FED examples for Gen LLM, while Action Relay uses ACT information to generate the screen-reader steps.For a font-size change, the steps select the Font Group Menu, Font Dropdown, and requested size.Source: §3.4.1 · p. 6, §3.4 · p. 6, §3.4.2 · pp. 6–7
• When a command has multiple plausible interpretations, Savant presents differing Gen LLM responses in ranked order for the user to disambiguate.Source: §3.4.3 · p. 7
• 78% accuracy (113 out of 145 NLCs) was achieved by Gen LLM on the ground-truth WOZ dataset.The evaluation compared each generated <CE, Value(CE)> pair with its ground-truth pair.Source: §3.4.4 · p. 7
• Commands combining multiple instructions, such as Copy and Paste in one command, were a primary source of incorrect Gen LLM outputs.The model was unable to process those composite commands and generate the action.Source: §3.4.4 · p. 7

4.1 Participants

• The 11 participants were blind screen-reader users who required screen readers, were proficient with JAWS, and had experience with Windows and its applications.The cohort’s average age was 44.2; five participants were male and six female.Source: §4.1 · p. 7

4.2 Study Design

• Participants completed representative tasks in six applications under JAWS-only, AccessBolt, and Savant conditions, with two applications per condition and counterbalanced order.The conditions used a within-subject design; AccessBolt presented visible controls in a pop-up, while Savant accepted natural-language commands.Source: §4.2 · p. 7

4.3 Apparatus

• The in-person study used a Windows laptop with the required applications and JAWS, plus an external QWERTY keyboard familiar to participants.Participants were provided a Lenovo ThinkPad.Source: §4.3 · p. 7

4.4 Procedure

• Participants practiced JAWS and the assistive systems for 20 minutes before completing tasks in counterbalanced order; tasks unfinished within 10 minutes counted as failures.Each study lasted nearly two hours and concluded with an exit interview.Source: §4.4 · pp. 7–8

4.5 Measurements

• Measurements covered task time, shortcut presses, 7-point SEQ ratings, NASA-TLX workload, and coded interview and utterance data.SEQ assessed perceived task difficulty; NASA-TLX measured perceived workload.Source: §4.5 · p. 8

4.6 Results

• Savant averaged 44 keys versus 163 with AccessBolt and 319 with JAWS; SEQ scores were 6.57, 5.6, and 2, and NASA-TLX scores 16.16, 28.93, and 49.9, respectively.The SEQ and NASA-TLX differences between successive conditions were statistically significant.Source: §4.6.3 · p. 9, §4.6.5 · pp. 9–10, §4.6.6 · p. 10
• Savant completed 18/22 tasks versus 14/22 with AccessBolt and 6/22 with JAWS, while average task time was 101 seconds versus 159 and 395 seconds, respectively.In JAWS, 8 of 16 failures involved locating controls hidden inside menus or containers; Savant’s few failures involved LLM command-mapping errors.Source: §4.6.1 · p. 8, §4.6.2 · p. 8, Figure 3 · p. 10
• All participants preferred Savant; 11/22 tasks succeeded on the first command, 7 after retries, and 4 remained incomplete, often because complex commands or speech-to-text errors impeded interpretation.Participants also valued reduced navigation and fewer key presses, while some requested composite commands and non-voice activation options.Source: §4.6.4 · p. 9, §4.6.7 · p. 10

5 DISCUSSION

• Savant’s user-study results indicate its potential to address interaction challenges arising from diverse application interfaces.The discussion frames this as potential rather than a conclusive resolution.Source: §5 · p. 11

5.1 Limitations and Future Work

• Unpredictable pop-ups, cross-window relationships, duplicate pop-ups, and context-dependent controls complicate support for pop-up and sub-window tasks.The authors identify these challenges as targets for future development.Source: §5.1.1 · p. 11
• Savant currently executes simple single-action commands; complex requests such as “Make all text italic” require decomposition into simpler actions.The proposed example breaks the request into clicking Select All and then Italic.Source: §5.1.2 · p. 11
• Tasks spanning applications require reasoning over multiple application contexts, identifying application-specific intent, and sequencing UI operations.The authors describe this as a topic for future research.Source: §5.1.3 · p. 11
• Chain-of-Thought prompting is proposed as a way to extend the method to complex sequential tasks by guiding the LLM to generate action sequences.The proposed prompt uses in-context few-shot examples.Source: §5.1.4 · p. 11
• Translation functionality is a proposed future extension for enabling support for multiple languages.The authors point to LLMs’ ability to understand and translate non-English instructions into English.Source: §5.1.5 · p. 11
• Wake-word activation was preferred by study participants, but implementation requires continuous microphone use and attention to detection, latency, privacy, and platform compatibility.The authors identify model training, optimization, and privacy management as challenges to address.Source: §5.1.6 · p. 11

6 CONCLUSION

• Savant uses LLMs to access and invoke desktop application controls on behalf of blind screen-reader users, with user-feedback-informed design showing improvements in interaction efficiency and usability.The authors describe this as progress toward a more inclusive digital environment for people with visual impairments.Source: §6 · p. 11

A.1 Example Prompt for the Seed LLM

• File Explorer’s (new item, folder) pair maps to “Create a new folder,” while WordPad’s (strikethrough, none) pair maps to striking selected text.Source: §A.1 · p. 14
• The prompt examples map Zoom’s start-meeting-with-video-off pair to a natural-language meeting request and Notepad’s edit-cut pair to “cut the selected text.”Source: §A.1 · p. 14
• For Word’s (share, copy link) pair, the example-generation output is “Share the document as a link.”Source: §A.1 · p. 14

A.2 Example Prompt for the Gen LLM

• The prompt pairs “Change the Margin to” commands for Narrow, Normal, Moderate, Wide, and Mirrored with corresponding (Margins, value) outputs.Source: §A.2 · p. 14
• For the user request “Set the Margin to Narrow,” the LLM-generated output is (Margins, Narrow).Source: §A.2 · p. 14

A.3.1 Participants

• The study recruited 11 screen-reader users proficient with JAWS and familiar with Windows applications; participants were almost evenly split by gender.The appendix reports an average age of 52.5 years (median 58, SD 15.1, range 31–73).Source: §A.3.1 · p. 14
• Participants reported frequent voice-assistant use: 10 used one at least daily, and Alexa, Siri, and Google Assistant were named by 9, 8, and 4 participants, respectively.Common uses included alarms and reminders, dictating messages and emails, phone calls, and information queries; demographic and usage information was self-reported.Source: §A.3.1 · p. 14, Table 3 · p. 15

A.3.2 Design

• Participants were given 26 tasks across Excel, File Explorer, Gmail, Word, and Zoom; tasks were selected based on closely related work, comparable in difficulty, and diverse in screen-reader UI interactions.Source: §A.3.2 · p. 14

A.3.3 Procedure

• The study used a Windows laptop with JAWS; participants received a demonstration and 10 minutes to learn the voice interface before application order was counterbalanced with a Latin square.Each session lasted nearly an hour, and participants received monetary compensation.Source: §A.3.3 · p. 14

A.3.4 Data collection

• Audio recordings were transcribed to collect participants’ natural-language commands for each task.Source: §A.3.4 · p. 14

• Table 5 lists the applications represented in the few-shot examples dataset.Source: Table 5 · p. 17