PaperlayerView exact revision on arXiv

Source-linked AI summary

LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents

Yogeswar Reddy Thota

arXiv:2606.30697v1cs.OScs.AIcs.CV

TL;DR

AI agents often interact with human-oriented operating systems through screenshots and OCR, despite existing accessibility structures that encode interface semantics. LUMOS exposes those structures as semantic blueprints, grounds actions through live UI state, and demonstrates the approach on simple text-entry and application-launch tasks. The paper therefore suggests semantic interaction layers as a path toward machine-readable interfaces for AI-native computing.

  • Problem

    Operating systems expose accessibility structures closer to AI agents’ needs than screenshots, but these structures are not designed as LLM-facing interaction layers.

  • Method

    LUMOS converts native and web UI semantics into compact blueprints, lets an LLM select constrained actions, and grounds interaction through live UI and pointer queries.

  • Results

    LUMOS is demonstrated on simple text-entry and application-launch workflows using visible UI interaction and re-observation.

  • Takeaways & Limitations

    The paper suggests that operating systems may need explicit machine-readable agent interfaces alongside human user interfaces.

  • Takeaways & Limitations

    The implementation is strongest on simple text-entry and launch tasks, while complex applications are not yet demonstrated as solved end-to-end.

Abstract

from arXiv · show

Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.

I. INTRODUCTION

LUMOS addresses the mismatch between human-optimized desktop interfaces and AI agents that need compact semantic state and grounded actions. It adds a machine-facing interaction plane using accessibility metadata, semantic pointer grounding, and accessibility-grounded observe–act loops without redesigning applications.

  • Motivation: Human-optimized desktop interfaces expose visual features, while AI agents need controls’ purposes, states, locations, and permissible actions.The paper contrasts desktop interaction with text-first and API-first environments where models can often act directly on symbolic text.
  • Problem: Screenshot-based computer-use agents must parse pixels, infer semantics, estimate coordinates, and ignore decoration, making open-ended desktop tasks expensive and brittle.Existing benchmarks show that open-ended computer tasks remain difficult for state-of-the-art agents, especially across desktop applications and operating-system surfaces.
  • Motivation: Operating systems and browsers already expose machine-readable interface structures containing names, roles, values, bounds, and supported actions.On Windows, UI Automation represents desktop elements as structured objects in a tree with properties including name, role, control type, value, bounding rectangle, and supported patterns.
  • Contribution: LUMOS transforms native or web UI structure into compact semantic blueprints, assigns stable identifiers, selects one valid action, executes it through visible mechanisms, and observes again.The layer adds a machine-facing interaction plane over human-first software without replacing the operating-system kernel or bypassing the visible interface.
  • Contribution: Live semantic pointer grounding queries the interface element under or near the cursor through UI Automation, including ElementFromPoint-style grounding.This provides cursor-position-aware semantic interaction based on live operating-system automation queries.
  • Contribution: Accessibility-grounded observe–act loops let agents navigate and control software through semantic roles, values, bounds, and structured actions instead of screenshot- or OCR-centric methods.The paper presents this as evidence that existing operating systems can expose a machine-readable interaction plane without requiring applications to be redesigned.

II. MOTIVATION: HUMAN-FIRST OS, MACHINE-FIRST AGENTS · III. FROM USER INTERFACES TO AGENT INTERFACES

Human-first interfaces hide system complexity in visual metaphors that suit people but force AI agents to spend resources extracting symbolic state from screenshots. LUMOS proposes a parallel, machine-readable agent interface that reuses operating-system and browser semantics to provide constrained, grounded interaction without requiring application redesign.

  • II. MOTIVATION: HUMAN-FIRST OS, MACHINE-FIRST AGENTS: Human-centered GUIs encode files, applications, and operations as icons, windows, buttons, menus, and gestures optimized for people rather than AI agents.
  • II. MOTIVATION: HUMAN-FIRST OS, MACHINE-FIRST AGENTS: Screenshots provide visual evidence, but agents must infer active applications, controls, text state, safe actions, and task completion instead of receiving directly symbolic information.This makes the model spend tokens and compute on perception that the operating system may already know.
  • II. MOTIVATION: HUMAN-FIRST OS, MACHINE-FIRST AGENTS: Operating-system and application frameworks can expose element bounds, accessible names, roles, states, and supported interaction patterns through automation interfaces such as UIA.These properties can be represented in a compact semantic blueprint for LLM use.
  • II. MOTIVATION: HUMAN-FIRST OS, MACHINE-FIRST AGENTS: LUMOS grounds the selected interaction and gives the LLM a safe, structured, visible, and reversible interface rather than unrestricted machine access.The system then performs the grounded interaction.
  • III. FROM USER INTERFACES TO AGENT INTERFACES: Computing interfaces have progressed through command lines, graphical interfaces, touch, and voice by changing how users access machine operations.
  • III. FROM USER INTERFACES TO AGENT INTERFACES: Agent interfaces form a parallel machine-readable plane that lets AI systems perceive state, understand actionable structure, and request constrained operations without replacing human interfaces.
  • III. FROM USER INTERFACES TO AGENT INTERFACES: The agent plane exposes semantics, accessibility trees, structured state, roles, values, action affordances, safety policies, and machine-readable completion feedback.It is coordinated with the human plane of windows, pixels, layout, pointing, keyboard, touch, and voice.
  • III. FROM USER INTERFACES TO AGENT INTERFACES: LUMOS prototypes this agent plane by reusing semantic metadata from operating systems and browsers, without requiring applications to be rewritten for AI agents.The paper positions it as a machine-native interaction layer for future AI-native computing environments.

IV. BACKGROUND · A. UI Automation and Accessibility Trees

UI Automation provides a programmatic, tree-structured representation of desktop interfaces, exposing element semantics and behavior through properties and control patterns. Although designed for accessibility rather than LLMs, UIA and analogous web structures already provide symbolic information that AI agents can use to interpret interfaces.

  • A. UI Automation and Accessibility Trees: UI Automation provides programmatic access to most desktop UI elements for assistive technologies and test scripts.It supports inspecting and manipulating interfaces.
  • A. UI Automation and Accessibility Trees: UIA organizes desktop elements in a tree rooted at the desktop, with application windows containing controls such as menus, buttons, edit fields, lists, and documents.This structure represents the containment relationships among interface elements.
  • A. UI Automation and Accessibility Trees: Each UIA element can expose properties and control patterns describing its semantics and behavior.These metadata describe what elements are and how they can operate.
  • A. UI Automation and Accessibility Trees: UIA was not designed for LLMs, but it already approximates a symbolic interface layer for AI agents.The framework provides machine-readable descriptions without requiring agents to infer all structure from pixels.
  • A. UI Automation and Accessibility Trees: UIA can identify visible regions as text boxes, buttons, list items, documents, or menus and provide names, values, focusability, and coordinates.These properties give agents semantic and interaction-relevant information about visible controls.
  • A. UI Automation and Accessibility Trees: Web interfaces offer analogous structure through DOM nodes and browser accessibility trees that describe a page’s functional organization.The same general principle extends beyond desktop UI Automation to browser interfaces.

B. LLM Agents and Computer Use · V. LUMOS ARCHITECTURE · A. Perception Layer

LLM agents must ground language instructions in interactive environments, while LUMOS provides a layered semantic observe–plan–act interface that uses native UI structure and compact machine-readable blueprints. Its perception layer extracts accessibility and browser information into actionable state for dynamic LLM control.

  • B. LLM Agents and Computer Use: LLM agents interleave reasoning with action selection, but benchmarks and recent work show that grounding instructions—especially across desktop applications—remains difficult.ReAct established interleaved reasoning and actions; WebArena, OSWorld, Mind2Web, and related work expose grounding challenges.
  • B. LLM Agents and Computer Use: LUMOS uses existing UI structure when available and reserves visual methods for semantic-API failures, reducing context size and improving action grounding.This positions LUMOS as complementary to screenshot- and vision-based computer-use systems.
  • B. LLM Agents and Computer Use: Unlike workflow-based robotic process automation, LUMOS exposes semantic observation and visible action primitives for an LLM to use dynamically.Traditional automation commonly relies on predesigned workflows or brittle UI scripts.
  • V. LUMOS ARCHITECTURE: LUMOS is organized as a layered observe–plan–act system in which the runtime observes semantic state, exposes universal visible actions, and lets the LLM choose each step.The default mode avoids dispatching a prewritten workflow.
  • A. Perception Layer: The perception layer queries Windows UIA trees or browser DOM/accessibility information and normalizes the active interface into a semantic blueprint.For native applications it reads foreground windows; for web pages it queries the browser session.
  • A. Perception Layer: Each blueprint can identify elements, controls, accessible names, current values, bounds, window and focus context, and goal-relevant semantic hints.Examples include identifiers such as A2 or W3 and roles such as button, document, edit, or menu item.
  • A. Perception Layer: The blueprint omits decorative visual details unless needed for action, making it cheaper to send to an LLM than a screenshot and easier to validate.Its compactness focuses semantic state on interaction-relevant information.

B. Live Semantic Pointer Grounding · C. Planner Layer

LUMOS grounds pointer interaction by querying operating-system UI elements and their semantics near screen coordinates, avoiding screenshot crops for target interpretation. Its LLM planner uses the goal, memory, and current blueprint to emit one constrained JSON action at a time, re-observing after each action.

  • B. Live Semantic Pointer Grounding: LUMOS queries the operating system for the UI Automation element at or near a screen coordinate instead of interpreting a cursor-centered screenshot crop.This approximates how humans use pointer position and vision together to identify the targeted object and possible action.
  • B. Live Semantic Pointer Grounding: The pointer query provides a live bridge between screen coordinates and the interface element associated with them.This replaces pixel-only interpretation with operating-system automation metadata.
  • B. Live Semantic Pointer Grounding: Pointer position becomes part of LUMOS’s semantic interaction plane, linking physical input coordinates to machine-readable interface semantics.The runtime can identify what element is under the cursor, what it means, and which operations it exposes.
  • B. Live Semantic Pointer Grounding: Live semantic grounding is useful when full-window blueprints are large, an element is ambiguous, or an agent must confirm a planned target.These conditions motivate querying a localized element near the cursor.
  • C. Planner Layer: The planner receives the user goal, recent memory, and current blueprint as inputs to its decision process.It is implemented as an LLM operating over the current semantic state.
  • C. Planner Layer: The planner must emit a single JSON action from a constrained schema rather than generating an unconstrained interaction sequence.This one-step discipline structures action selection around the next operation.
  • C. Planner Layer: One-step planning avoids hallucinated long scripts and requires the agent to re-observe after each action.The paper compares this feedback pattern to a human checking the screen after clicking or typing.

D. Universal Action Schema

LUMOS defines a small, application-neutral action schema built from visible UI primitives for observing controls, launching applications, manipulating elements, entering text, using keyboard input, and explicitly finishing tasks.

  • Universal Action Schema: LUMOS exposes visible UI primitives for observation, search-overlay access, safe application launching, element interaction, text entry, keyboard input, and task completion.The primitives include observe, open_windows_search, open_app, click, double_click, drag, type_text, set_text, press_key, and finish.
  • Universal Action Schema: The schema distinguishes appending text with type_text from replacing existing text with set_text.type_text targets a focused or targeted control, while set_text replaces existing text rather than appending.
  • Universal Action Schema: The finish primitive explicitly stops execution when the goal is satisfied.This makes task completion an explicit action in the schema.
  • Universal Action Schema: Application-neutrality represents goals such as drafting an email as visible UI actions over observed controls rather than backend mail API calls.The schema operates through visible interface interactions instead of application-specific backend APIs.

E. Memory and Repair Layer · F. Safety Layer · VI. PROTOTYPE IMPLEMENTATION

LUMOS combines action memory, repair guidance, confirmation-gated safety controls, and a Python/Windows prototype that grounds model-selected actions in semantic observations. Its default runtime re-observes after actions, exposes universal primitives, and keeps task strategy with the LLM rather than fixed workflows.

  • E. Memory and Repair Layer: The memory layer tracks recent actions, failures, and entered text to prevent repeated actions and guide finishing or replacing text.It also stabilizes Windows Search handoffs by ensuring a pending query is typed into Search.
  • F. Safety Layer: LUMOS constrains planning with an allowlist and confirmation policy for potentially risky operations such as sending email, deleting data, or destructive hotkeys.These operations require explicit confirmation.
  • F. Safety Layer: System settings are accessed through visible settings UI rather than hidden backend APIs, preserving behavior like a visible user instead of an unrestricted system process.
  • VI. PROTOTYPE IMPLEMENTATION: The prototype uses Python on Windows, combining UI Automation for native desktop observation with browser automation for web page structure.It supports local or compatible LLM backends and maintains persistent browser sessions, native blueprints, and a shared ID map.
  • VI. PROTOTYPE IMPLEMENTATION: Opt-in deterministic scaffolds support reproducible demos on slow local hardware, but they are disabled by default and treated as ablation knobs.The paper’s architecture and case-study claims refer to the default observe–LLM–act mode.
  • VI. PROTOTYPE IMPLEMENTATION: The implementation rules assign task strategy to the LLM and expose only universal perception and action primitives.
  • VI. PROTOTYPE IMPLEMENTATION: The system re-observes after actions rather than assuming success, while the runtime can repair invalid syntax, prevent repeated launches, and convert appending into replacement.Higher-level sequencing remains a planner decision over the current blueprint, and sending email requires explicit approval.

VII. CASE STUDIES · A. Opening Notepad and Writing Generated Text · B. Windows Search Handoff for an Outlook Query

The case studies show LUMOS handling generated text through explicit completion actions and launching unavailable desktop applications through a visible Windows Search handoff. These workflows preserve user intent while constraining text entry, state completion, and query submission.

  • A. Opening Notepad and Writing Generated Text: LUMOS distinguishes literal text-entry goals from generated-text goals when writing content in Notepad.For generated-text goals, the model produces the content itself while LUMOS tracks whether it has already been entered.
  • A. Opening Notepad and Writing Generated Text: Generated-text goals require the model to produce the requested content rather than type the instruction verbatim.The example instruction asks for a short essay about AI in three paragraphs.
  • A. Opening Notepad and Writing Generated Text: LUMOS treats stopping as an action by making completion part of the action schema.This lets the model declare that the visible state satisfies the user goal and prevents continued revising, retyping, or appending.
  • B. Windows Search Handoff for an Outlook Query: When desktop applications are unavailable by executable name, LUMOS follows the visible workflow of opening Windows Search, entering the app name, and pressing Enter.The workflow mirrors how a human would launch such an application.
  • B. Windows Search Handoff for an Outlook Query: The runtime opens Windows Search and preserves the pending query after the model emits an open_windows_search action.If the next observation still shows Search, the system ensures the pending handoff continues.
  • B. Windows Search Handoff for an Outlook Query: The handoff types and submits “outlook” before the agent proceeds, preventing stale web context from hijacking the next action.This preserves LLM intent but is a launch-handoff mechanism, not evidence that Outlook composition workflows have been fully solved.

VIII. EVALUATION PLAN · IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE

The paper proposes evaluating semantic grounding against screenshot/OCR interaction while using regression tests to validate prototype behavior. It frames accessibility APIs as machine-readable cognitive infrastructure that can support AI agents alongside human users.

  • VIII. EVALUATION PLAN: The evaluation plan compares screenshot+OCR+LLM with blueprint+LLM pipelines on identical tasks using success, latency, tokens, observation size, and recovery turns.It also proposes measuring blueprint compression.
  • VIII. EVALUATION PLAN: Prototype regression tests cover schema coercion, generated-text handling, text replacement, Search handoff, safety checks, and blueprint refresh behavior.They do not replace human-subject or benchmark evaluation, but expose architectural behavior and common model-error repairs.
  • IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE: Accessibility APIs expose roles, labels, values, states, selection, focus, and interaction patterns beyond pixels, and LUMOS repurposes this infrastructure for AI cognition.These APIs were originally developed for people with diverse perceptual and motor abilities.
  • IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE: LUMOS treats Notepad failures as semantic bridge problems requiring literal-copy rejection, replacement instead of appending, repeat guards, explicit completion, and stable multiline entry.The diagnostic evidence includes instruction copying, append-style corrections, repeated fragments, and character-level typing problems.
  • IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE: The Notepad diagnostic counts summarize development-log failure and repair signals rather than benchmark results.They motivate literal-copy rejection, append-to-replace repair, repeat guards, and explicit completion handling.
  • IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE: Accessibility metadata forms a machine-readable contract linking semantic identity, editable state, meaningful operations, and physical coordinates.Examples include accessible button names, text-field value providers, control patterns, and bounding rectangles.
  • IX. ACCESSIBILITY APIS AS COGNITIVE INFRASTRUCTURE: Accessibility infrastructure may become the foundational semantic substrate of AI-native operating systems while continuing to support human-centered visual interaction.The proposed model exposes parallel planes for human perception and machine cognition.

X. DISCUSSION · A. Why This is an Operating-System Problem · B. Why Not Only Screenshots?

LUMOS frames AI interaction as an operating-system problem, using accessibility APIs as an initial AI-native interaction plane and envisioning richer machine-readable OS contracts. It also argues that semantic blueprints should complement screenshots by separating structured-state provision from LLM planning.

  • A. Why This is an Operating-System Problem: LUMOS treats accessibility APIs as the first version of an AI-native interaction plane.Current desktops offer human-facing interfaces alongside accessibility APIs for assistive technologies.
  • A. Why This is an Operating-System Problem: AI agents need an operating-system-level interface designed for machine cognition.The paper identifies this as LUMOS’s long-term implication.
  • A. Why This is an Operating-System Problem: Future operating systems could expose application intentions, commands, reversible operations, security boundaries, approval requirements, and task progress.These capabilities would provide richer semantic state directly to agents.
  • A. Why This is an Operating-System Problem: An OS could provide a trusted machine-readable contract describing what is visible, actionable, and safe.This would reduce the need for AI systems to infer everything from pixels.
  • B. Why Not Only Screenshots?: Screenshots remain valuable when applications expose little useful semantic structure.The paper does not claim semantic interaction should replace visual agents in such cases.
  • B. Why Not Only Screenshots?: Semantic blueprints separate OS-supplied structured state from LLM planning instead of combining perception and action grounding.This separation is described as easier to test, cheaper to prompt, and more aligned with security constraints.

C. Why Not Only APIs? · XI. LIMITATIONS · XII. CONCLUSION

LUMOS favors inspectable, reversible visible-UI actions over hidden APIs when user expectations or safety require interface-level interaction. Its prototype demonstrates a semantic architecture but remains limited by imperfect UI semantics, dynamic interfaces, action-selection errors, and unresolved security risks.

  • C. Why Not Only APIs?: LUMOS favors visible UI actions because they are inspectable and reversible, rather than bypassing the interface through hidden APIs.For “draft an email, do not send it,” the model should fill the visible draft and stop before Send.
  • XI. LIMITATIONS: LUMOS depends on exposed UI semantics, which may be incomplete, ambiguous, duplicated, custom-rendered, or unstable between observation and action.These conditions can cause incorrect actions, misunderstood completion, or multiple recovery turns.
  • XI. LIMITATIONS: Security remains central because an AI-controlled UI layer must prevent unintended submission, deletion, credential exposure, and privilege escalation.The limitation is explicitly tied to preventing harmful UI outcomes.
  • XI. LIMITATIONS: The prototype does not claim human-level autonomy; opening Notepad and typing text demonstrates the interaction model rather than solving all desktop workflows.The research value is attributed to semantic extraction, grounded action IDs, constrained universal actions, memory, safety, and explicit completion.
  • XI. LIMITATIONS: The implementation is strongest on simple text-entry and launch tasks, including opening Windows Search, submitting an application query, and re-observing the resulting UI.The remaining workflow depends on the quality of the next blueprint and the LLM’s visible-action choices.
  • XI. LIMITATIONS: Complex applications such as video editors, mail clients, and custom-rendered professional tools are not yet demonstrated as solved end-to-end.The stated reason is limitation by subsequent blueprint quality and correct visible-action selection.
  • XII. CONCLUSION: LUMOS uses existing accessibility and UI automation substrates to expose machine-readable blueprints, support LLM planning, ground actions through live UI state and pointer semantics, and execute constrained visible-UI operations.The conclusion presents this as a practical semantic interaction layer for human-first operating systems.
  • XII. CONCLUSION: The approach occupies a promising middle ground between screenshot-heavy agents and brittle task scripts.This positioning is stated in the conclusion as a characterization of LUMOS’s practical semantic interaction layer.
Loading 2606.30697v1…