AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

• Multiple sequential UI steps increase latency, token and monetary costs, while long interaction chains compound wrong-control selections and hallucinations into task failures.UI agents must repeatedly reason over large volumes of interface state, and unfamiliar application UIs remain difficult to generalize to.Source: §1 · pp. 1–2
• AXIS is an API-first framework that explores applications, learns from support documents and action trajectories, and constructs new APIs when needed.Regular UI interactions are retained only when related APIs are unavailable.Source: §1 · p. 2
• AXIS replaces unnecessary multistep UI operations with API calls, enabling fewer tokens and more accurate, code-formatted responses with low latency and high reliability.Figure 1 contrasts sequential UI-agent interactions with AXIS completing a task through a single API call.Source: §1 · p. 2, Figure 1 · p. 2
• The framework proposes a HACI paradigm in which applications can be wrapped with APIs and simplified UIs to become autonomous agents, supporting a path toward an Agent OS.The paper positions this design as a response to applications built primarily for human-computer interaction.Source: §1 · pp. 2–3
• Experiments on Microsoft Word tasks evaluate whether LLM-based agents reduce cognitive load, improve efficiency, and differ from UI agents in user experience.Source: §1 · p. 3

2.1 LLM-based UI Agent

• 2.1 LLM-based UI Agent: MLLM-powered agents operate mobile, web, and operating-system applications, but human-oriented interfaces make repetitive tasks inefficient.Source: §2.1 · p. 3

2.2 Agent Operating System

• 2.2 Agent Operating System: Existing Agent OSes decompose tasks across applications but still depend on inefficient clicking and swiping that can take control away from users.Source: §2.2 · p. 3

2.3 UI design in LLM era

• 2.3 UI design in LLM era: AXIS explores applications to identify UIs that can be replaced by API calls, adapting design from HCI toward HACI.Source: §2.3 · p. 3

• AXIS defines an environment as interactive entities, primarily Windows applications such as Microsoft Word, exposed through state() and step() interfaces.Source: §3 · p. 3
• A skill is a structured, high-level unit for a specific task, consisting of a description, executable skill code, and usage example, with API actions prioritized.Source: §3 · p. 4
• Skills span atomic, composite, and hybrid UI/API types and can be nested into hierarchies, such as a depth-3 chain of skills A, B, and C.Source: §3 · p. 4

4 Design of AXIS

• AXIS consists of trajectory collection, skill generation, and skill validation, with validation reducing hallucination and preserving skill generalizability before library insertion.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Stage I: Trajectory Collection

• 4.1 Stage I: Trajectory Collection: Follower Mode follows documented tasks, while Explorer Mode generates instructions and varies states, directions, and proficiency levels.Source: §4.1 · p. 4
• The framework prioritizes API-only representations when a skill can be expressed through either UI or API actions.Source: §4.1 · p. 4

4.2 Stage II: Skill Generation

• 4.2 Stage II: Skill Generation: A monitor extracts trajectory insights, a generator produces executable code with placeholders, and a translator uses documentation and the skill library to convert UI segments into APIs.Source: §4.2 · pp. 4–5

4.3 Stage III: Skill Validation

• 4.3 Stage III: Skill Validation: Static checks verify code structure and references, while dynamic validation tests generalizability in the real environment and evaluates final states.Source: §4.3 · p. 5

5.1 SKill Exploration

• AXIS discovered 73 Word skills from 347 seed files, including 44 hierarchy-1, 24 hierarchy-2, 3 hierarchy-3, and 2 hierarchy-4 skills.Source: §5.1 · p. 6

5.2 Task Completion

• AXIS completed 50 Word tasks in 29.9 seconds on average versus 59.5 seconds for the UI Agent, while achieving higher success rates and lower costs.Source: §5.2 · p. 6, Table 1 · p. 6
• AXIS predominantly used integrated API skills, reducing UI actions through skill-based abstraction and streamlined task execution.Source: §5.2 · p. 6
• AXIS used API actions at a 55.7% rate, including 23.1% advanced APIs, compared with 8.1% for the UI Agent.Source: §5.2 · p. 6, Table 2 · p. 6

6.1 Experimental Metrics

• The study measured NASA-TLX dimensions, learning effort, agent fluency and reliability, UI dependency, decision consistency, completion speed, time, success, steps, and backend costs.Source: §6.1 · p. 7

6.2 Results

• Agent-assisted methods reduced mental and physical demand and frustration relative to manual work, especially for complex L2 tasks (p<0.05).Source: §6.2 · p. 7
• AXIS significantly outperformed manual and UI-agent methods in time efficiency, particularly for complex L2 tasks, while maintaining near-human accuracy.Source: §6.2 · p. 7
• LLM-based agents reduced learning effort, with the strongest reduction for difficult tasks.Source: §6.2 · p. 7
• Participants preferred AXIS for speed, fluency, and reliability, while its decisions aligned better with human reasoning on complex L2 tasks.Source: §6.2 · p. 7

• AXIS prioritizes API calls over UI interactions to reduce inefficiencies and cognitive burdens in complex MLLM-agent tasks.Source: §7 · p. 7
• Task completion time fell by 65%-70% and cognitive workload by 38%-53%, while AXIS maintained human-level accuracy in Microsoft Word experiments.Source: §7 · p. 7
• The framework suggests that API-first agents can streamline interactions, reduce latency, and improve reliability for faster task execution.Source: §7 · p. 8

• Datasets were collected internally or obtained externally with no infringement on individual or organizational rights, and participants volunteered and received compensation.Source: §8 · p. 8

9 Limitations

• AXIS primarily relies on Python-based APIs, limiting support for applications without native Python interfaces.Source: §9 · p. 8
• Its exploration process still requires improvements in stability and efficiency before AXIS can achieve an Agent OS.Source: §9 · p. 8

A Interfaces in Environment

• The environment exposes state() for environment information and step() for executing agent actions through a skill executor.Source: A Interfaces in Environment · p. 12

B.1 Skill Component

• Each skill contains code, a functional description, and usage examples that help the agent select skills and format invocation parameters.Source: §B.1 · p. 12

B.2 Skill Type

• AXIS supports five skill types: atomic UI, atomic API, composite UI, composite API, and API-UI hybrid skills.Source: §B.2 · p. 12

B.3 Initial Skill Repository

• The initial skill repository is built from basic actions derived from UFO before additional skills are explored and developed.Source: §B.3 · p. 12

B.4 Skill Translation Example

• Skill generation and translation examples illustrate how AXIS converts interaction trajectories and UI operations into structured, API-oriented skills.Source: §B.4 · p. 12, Table 6 · p. 13, Table 7 · p. 14

B.6 Skill Dynamic Validator

• Dynamic validation tests generated skills in randomized Word documents and adds them to the library only after execution and final-state evaluation succeed.Source: §B.6 · p. 16

C.1 Experiment Procedure

• The study used five Microsoft Word tasks classified as low-difficulty L1 or high-difficulty L2 according to interaction complexity.Source: §C.1 · p. 16
• Participants completed tasks manually and with both UI Agent and AXIS assistance through a web interface, with execution processes recorded.Source: §C.1 · p. 16

C.2 Participants Recruitment

• Twenty participants aged 18 to 40, all with some Microsoft Word experience, took part and received 50 CNY compensation for an approximately 30-minute experiment.Source: §C.2 · p. 16

C.4 User Study Tasks

• Five Microsoft Word tasks were sampled and classified into low difficulty (L1) and high difficulty (L2) levels using the feasibility-study criteria.The tasks were drawn from official Word documentation and GPT-generated results.Source: §C.4 · p. 16

C.5 User Study Survey Form

• Participants completed tasks in manual and agent modes, with agent-mode tasks submitted to two agents for automatic completion.The agent-mode interface displayed the original and completed pages along with each agent’s decision-making process.Source: Figure 7 · p. 17, Figure 5 · p. 17, Figure 6 · p. 17

C.5.1 Cognitive load related forms

• NASA-TLX and learning-effort surveys measured cognitive load after participants completed tasks in both modes.The study also collected surveys on perceived speed, fluency, reliability, decision consistency, and UI dependency.Source: §C.5.1 · p. 18, §C.5.2 · p. 18

C.5.2 Human Preference related forms

• Human-preference questionnaires assessed perceived speed, fluency, reliability, decision consistency, and UI dependency after agent-mode use.These forms complemented the cognitive-load measures with subjective assessments of agent interaction.Source: §C.5.2 · p. 18

D Feasibility Study

• The feasibility study sampled 50 Microsoft Word tasks and classified difficulty using required UI interactions, maximum control depth, and ribbon switches.An L2 task must satisfy all three thresholds: ≥3 UI interactions, maximum control depth > 2, and ≥1 ribbon switch.Source: D Feasibility Study · p. 18
• Table 12 reports the distribution of required execution steps and difficulty levels across the 50 feasibility-study tasks.Execution steps represent the number of UI actions a human would typically perform.Source: D Feasibility Study · p. 18

E.1 AXIS help to digest unnecessary Application UIs

• AXIS uses LLM-powered self-exploration to identify application controls that can be converted into APIs.The exploration targets existing API and UI functions and can uncover redundant UI designs.Source: §E.1 · pp. 18, 22
• The study materials include surveys for NASA-TLX, learning effort, perceived speed, UI dependency, and decision consistency, fluency, and reliability.These instruments provide the user-study context for evaluating interaction with different agents and modes.Source: Figure 8 · p. 19, Figure 9 · p. 20, Figure 10 · p. 20
• In the UI hierarchy, red nodes are API-ifiable controls, while blue nodes are controls that lack a corresponding triggering API or are difficult to describe in language.Higher-level UI elements are parents of lower-level child elements in the hierarchy.Source: §E.1 · p. 22
• A node is non-essential when it and all of its child nodes can be API-ified.The criterion is represented as true when the node and all children are red, and false otherwise.Source: §E.1 · p. 22
• AXIS can guide application providers to simplify or eliminate non-essential UI elements while preserving their functions through API calls.This reframes UI design for an agent-based system and supports the proposed Agent OS direction.Source: §E.1 · p. 22

E.2 Turn An Application into an Agent

• AXIS is adaptable to new applications with basic API and documentation support, using operational manuals plus Environment State and Basic Action Interfaces.The Environment State Interface provides environment information, while the Basic Action Interface supports basic interactions.Source: §E.2 · p. 22
• AXIS can automatically and continuously explore applications, discover new skills, extend functionality, and integrate across software environments through API-driven interactions.The framework is presented as a way to enhance functionality and user experience.Source: §E.2 · p. 22

F.1.1 Follower Agent

• The Follower Agent is a Windows UI-focused assistant that selects available controls and takes one-step actions based on current application state, prior plans, user-request history, and previous function returns.Each one-step action calls a function once, while the agent may revise its prior plan using updated control information.Source: §F.1.1 · p. 22
• The Follower Agent returns structured JSON containing observations, reasoning, selected control labels and text, an API function with arguments, task status, a plan, and review.It must select controls from the provided list, match control text exactly, and use CONTINUE, FINISH, or ERROR status values.Source: §F.1.1 · pp. 23–24
• The agent reviews action history and current state to verify effects, prioritizes general functions, avoids redundant or completed actions, and stops with FINISH when the request is complete.If a previous action failed to take effect, it may retry or rectify it and should continue; missing required actions must be added to the plan.Source: §F.1.1 · pp. 24–25
• The Follower Agent executes requests by grounding plans and API arguments in the observed application state, strictly following the request and available functions.It must avoid adding unrequested actions, use only listed controls and functions, and output parseable JSON.Source: §F.1.1 · p. 25

F.1.2 Explorer Agent

• The Explorer Agent learns application skills through one-step actions on annotated controls, using screenshots, control states, action histories, and available APIs.Exploration is restricted to provided controls and techniques, with outputs specifying the observation, selected control, API function, arguments, step, and action.Source: §F.1.2 · pp. 26–27
• The Explorer Agent switches between deep and broad control-tree exploration: True dives into descendants, whereas False explores controls at the same or higher level.The dive-into strategy determines whether exploration proceeds deeply from the current control or widely across its level.Source: §F.1.2 · p. 27
• The Explorer Agent must inspect both screenshots and control lists, avoid repetitive historical actions, and navigate through other controls when the required control is unavailable.Selected control labels and text must match the provided list exactly, while plans describe only future steps after the current action.Source: §F.1.2 · p. 27
• The Explorer Agent compares screenshots with the previous step, avoids repeating completed actions, and saves screenshots only when they are strongly useful for future steps.It must provide a reason when saving and review prior trajectories to avoid duplicating actions.Source: §F.1.2 · pp. 27–28

F.2.1 Generator Agent

• The Generator Agent creates skill code from a skill description and logical flow, with each skill defined as a function that interacts with the desktop application.It may combine existing skills and should follow their descriptions and examples when generating new code.Source: §F.2.1 · p. 28
• Generated skills use Python with a consistent executor,args_dict interface, support locating controls by UUID or arguments, and return JSON containing thought, code, description, and example fields.The code must use the prescribed executor methods and include documentation and clear calling parameters.Source: §F.2.1 · pp. 28–29
• The Generator Agent’s dynamic skill execution supplies control identifiers or names through args_dict so generated logic can locate the target control item.The control lookup uses get_target_control_by_args.Source: §F.2.1 · pp. 29–30

F.2.2 Translator Agent

• The Translator Agent converts skill code based on UI control actions into equivalent Python API calls using the win32com library.It receives skill descriptions, logic, original code, current skills, and available APIs, and must manipulate the document object directly without navigating the UI.Source: §F.2.2 · p. 30

F.3.1 Validator Agent

• The Validator Agent proposes a specific task and precise function arguments from the function and document environment to test function accuracy.It uses function code, descriptions, examples, canvas and control states, and screenshots, then returns the observation, task, thought, function, and args in JSON format.Source: §F.3.1 · p. 31

F.3.2 Evaluator Agent

• The Evaluator Agent judges whether an agent completed the original request using its thought, execution trajectory, environment states, canvas difference, and before-and-after screenshots.Its inputs include initial and final control states as well as the final canvas state.Source: §F.3.2 · p. 32
• The evaluator compares screenshots and canvas differences while reviewing intermediate execution steps to determine whether the task was completed.Canvas differences are represented as changes in XML derived from the document object model.Source: §F.3.2 · pp. 32–33
• The agent uses the available application APIs and execution trajectory items, where each action records an API call used to interact with the application window.The evaluator receives the original request, thought, trajectory, canvas diff, initial and final control states, and final environment status.Source: §F.3.2 · pp. 32–33
• The evaluator models actionable controls including buttons, editable fields, tabs, lists, and menus, with each control’s text, type, selection status, and child controls recorded.Source: §F.3.2 · p. 32
• The evaluator outputs a strict JSON response containing a task-completion value of yes, no, or unsure and a detailed completion judgment.The required fields are task_complete and complete_judgement.Source: §F.3.2 · p. 33
• The evaluator checks final control and environment states, trajectory consistency, and whether expected controls or canvas content changed as required.An empty action also causes the task to be judged incomplete.Source: §F.3.2 · p. 33