PaperlayerView exact revision on arXiv

Source-linked AI summary

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

Rui Xie, Lu Chen

arXiv:2608.26991v1cs.AI

TL;DR

Software-operating agents face an interface mismatch because screenshots incompletely expose software state and GUI actions are brittle and low-level. ASIL replaces screenshot-and-click with structured JSON observations and code-executable semantic actions realized through feasible file, script, or service paths. Across the benchmark, ASIL reports strong results with short trajectories and supports substantial SFT and RL gains, while its claims remain bounded by perceptual tasks, access-path eligibility, and comparison conditions.

  • Problem

    Screenshots expose only a partial software-state projection, while GUI events are brittle and semantically weak for long-horizon software operation.

  • Method

    ASIL exposes structured JSON observations and code-executable semantic actions through the deepest feasible file, scripting, or service access path for each application.

  • Results

    ASIL achieves strong inference results with short semantic trajectories and yields double-digit training gains on Qwen3.5-2B and Qwen3.5-9B through SFT and on-policy RL.

  • Takeaways & Limitations

    Software operation is best treated as interaction over state and verifiable artifacts rather than pixels and motor traces, within software exposing open access paths.

  • Takeaways & Limitations

    Perceptual tasks remain constrained: on the hard-suite GIMP Creative region, GPT-5.4 ASIL averages 14.7 and records zero full passes across 24 tasks.

Abstract

from arXiv · show

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

1 Introduction

The paper frames GUI operation as an interface mismatch: screenshots expose incomplete software state and GUI events impose brittle, low-level control. ASIL replaces this loop with structured observations and semantic actions, then evaluates the approach across applications, tasks, baselines, and training settings.

  • Motivation: GUI agents inherit a human-native screenshot-and-click loop that omits hidden software state and expands semantic goals into brittle low-level events.Screenshots expose only a visible projection, while GUI actions require repeated visual inference and motor sequences.
  • ASIL: ASIL defines an agent-native interface using structured JSON software-state observations and code-executable semantic operations.The interface can use file formats, scripting runtimes, or service APIs while preserving the same agent form.
  • Evaluation: ASIL is instantiated across 15 applications with 300 single-application and 80 multi-application benchmark tasks using shared task definitions and validators.The benchmark supports direct ASIL/GUI comparison and includes both single-application and multi-application workflows.
  • Evaluation: ASIL adds repaired GUI and native-interface baselines, validity checks, and training studies to bound and extend the interface comparison.The reported training results include SFT and resource- and time-limited on-policy RL on Qwen3.5-2B and Qwen3.5-9B.

2 Related Work

Related work shows that agents can operate software through screenshots, GUI events, tools, APIs, and executable code. ASIL builds on the code-agent precedent while asking whether screenshot-and-click should remain the default interface for software operation.

  • GUI agents: GUI-agent research typically frames software use as interpreting rendered screens and emitting clicks, keystrokes, drags, or other human-like operations.These systems demonstrate real-software operation through human-facing interfaces.
  • Hybrid and API interfaces: Recent systems add OS skills, hybrid GUI–API control, dynamically generated programmatic actions, or API routing to improve GUI-agent capability and efficiency.The cited approaches include OS-Copilot, UFO2, DynaSaur, AXIS, and API-based web agents.
  • Code-agent precedent: SWE-agent and CodeAct establish purpose-built interfaces and executable Python as precedents for agent interaction with readable state and tools.Claude Code and Codex extend this pattern to environments with executable commands and verifiable feedback.
  • ASIL’s position: ASIL transfers the code-agent interface lesson to existing GUI software while studying whether screenshot-and-click should remain the default substrate.Its focus is an interface question rather than another screenshot-grounding improvement.

3 Interface Mismatch and ASIL

The paper characterizes screenshot-and-click as inefficient because pixels hide task-relevant state and GUI events lengthen action sequences. ASIL instead uses structured state and schema-constrained semantic actions within an observe–act–check loop.

  • Interface mismatch: Every screenshot-and-click step incurs visual encoding and multimodal reasoning, while low-level events expand semantic goals into long trajectories.Long trajectories increase inference-time latency and make training rollouts more expensive.
  • Structured observation: ASIL’s OBSERVATION object organizes task metadata, application state, interactive elements, environment context, navigation structure, and a textual summary.This representation preserves software information that screenshot-based agents must otherwise recover.
  • Design principles: ASIL follows four principles: completeness, semanticity, stability, and composability.Together they expose relevant state, align actions with meaningful operations, preserve valid targets, and express complex tasks with fewer higher-level actions.
  • Formal interface: ASIL substitutes the pixel/event interface (Pix, M_k) with structured observation Φ and semantic action A in one observation–action–transition loop.The formalization treats S as latent software state, O as structured observations, and A as schema-constrained semantic actions.
  • Formal interface: One semantic action can realize the same transition as a sequence of k GUI events with k ≫ 1, shortening trajectories and the RL credit-assignment horizon.The observation mapping is engineered to preserve task-relevant state beyond the rendered viewport.

4 Realizing ASIL in Real Software

ASIL is realized by mapping heterogeneous applications to a shared observe–execute–validate contract through the deepest feasible open access path. The benchmark uses common tasks and validators across 15 applications, while the training pipeline reuses verified ASIL trajectories.

  • Realization methodology: The ASILization pipeline selects the deepest feasible access path exposing stable state and semantic operations close to an application’s transition system.The adapter then presents heterogeneous software through a common contract.
  • Realization patterns: ASIL uses file-backed, native-scripting, and service/API realizations as pathways to one normalized JSON representation.Across 15 implementations, six are file-backed, four native-script, and five service/API realizations.
  • Eligibility and onboarding: Applications qualify only when they expose an open read path and a semantic-action path; opaque software without parseable or structured access remains out of scope.This bounds coverage to software with usable file, scripting, structured-command, or service surfaces.
  • Benchmark construction: The benchmark contains 300 single-application tasks across 15 domains and 80 multi-application workflows requiring artifacts or information to cross application boundaries.The coverage includes creative, productivity, service-backed, code, workspace, and desktop-utility environments.
  • Benchmark construction: The same task definitions, initial artifacts, state validators, and result directories are used across ASIL and GUI modes.Final software state, rather than surface-level action histories, determines success.
  • Training pipeline: ASIL training uses evaluator-backed verified trajectories for supervised and reinforcement-learning studies.The same runtime supplies replay-based SFT data and RL rewards.

5 Experiments

Across benchmark, native-interface, training, and ablation experiments, ASIL consistently supports shorter and stronger software operation than repaired screenshot-and-click control, while exposing residual coverage limits on difficult visual workflows. Its structured traces also support effective small-scale SFT and resource-limited RL across models and task regimes.

  • Main benchmark: 81.6 versus 6.6 and 81.2 versus 26.6 strict success show ASIL’s large aggregate advantage over repaired GUI / 50max control for GPT-5.4 and sonnet4.6.ASIL averages fewer than five executed actions under a 15-step budget, while GUI uses a 50-step budget.
  • Main benchmark: 15.0 versus 6.6 and 53.3 versus 26.6 strict success on easy60 narrow, but do not eliminate, the repaired GUI gap.The authors treat easy60 as an OSWorld-comparable reference band rather than an exact difficulty match.
  • Native baselines: ASIL exceeds LibreOffice UNO by 28–38 strict points on 60 matched tasks, while matching draw.io’s MCP content contract for GPT-5.4 and trailing it for sonnet4.6.The comparison supports a compositional contract rather than universal per-application dominance.
  • Training: 72.1 from 58.0 and 80.4 from 66.6 after SFT rise further to 74.4 and 82.2 with resource- and time-limited on-policy RL for Qwen3.5-2B and Qwen3.5-9B.The training loop reuses verified ASIL observations, actions, artifacts, and evaluator outcomes; the setup uses thousands of SFT samples and 320 RL training tasks.
  • Hard-task evaluation: +17.1 and +19.8 points are the Qwen3.5-9B SFT and RL gains over base on the held-out 80-task hard suite, exceeding the corresponding +13.8 and +15.5 main-benchmark gains.Multi-App rises from 33.9 to 73.9 to 77.2 across Base, SFT, and RL.
  • Failure analysis: 15.2 is the strongest-row average on GIMP-heavy creative real-photo tasks, where pixel-level image semantics exceed the current semantic action vocabulary.The remaining ASIL failures are attributed to residual modeling and coverage limits, including incorrect action choice and underrepresentation.

6 Conclusion

ASIL reframes GUI software operation as interaction over structured state and verifiable artifacts rather than pixels and motor traces. Across a 380-task, 15-application benchmark, it combines strong inference performance with short semantic trajectories and training gains, while its comparisons remain deliberately bounded.

  • ASIL replaces screenshot-and-click with structured software state and code-executable semantic actions realized through each application's deepest feasible access path.
  • On a 380-task benchmark across 15 applications, ASIL obtains strong inference results with short semantic trajectories and supports practical training with small-scale SFT and on-policy RL.
  • Repaired 50-step GUI control reaches 6.6 and 26.6 strict success on the full benchmark, versus ASIL's stronger results in the same comparison.
  • Against native interfaces, ASIL clearly exceeds LibreOffice UNO but only matches draw.io MCP, bounding the comparative claim.

Limitations

The paper identifies four limitations: comparison asymmetry, small-model RL instability on long-horizon tasks, incomplete coverage of opaque applications, and intrinsically perceptual success criteria. These boundaries constrain how broadly ASIL's results should be interpreted.

  • The original comparison does not isolate the interface variable because ASIL receives evaluator-derived success hints while GUI prompts do not.The hint effect is measurable, and broader independent validation across all 380 tasks remains future work.
  • On the 80-task hard suite, Qwen3.5-2B gains shrink to +6.0 with SFT and +3.6 with RL, with the selected RL checkpoint trailing SFT by 2.4 points.The paper attributes this to the small-model RL operating point rather than an interface problem.
  • ASIL's current realization covers software with at least one open access path, leaving fully opaque applications for future work.The excluded applications are closed-source, difficult to unpack, and lack rich external scripting or service interfaces.
  • Tasks dominated by aesthetic or perceptual criteria remain constrained because the current semantic action vocabulary cannot fully express them.In GIMP real-photo editing, GPT-5.4 ASIL averages 14.7 and records zero full passes across 24 tasks; screenshots alone do not fix this.

Ethical Considerations

ASIL is an interface and evaluation framework rather than a user-facing autonomous deployment. The paper scopes its claims and recommends safeguards for deployments connected to real accounts or files.

  • ASIL is an interface and evaluation framework, not a user-facing autonomous deployment.
  • Deployments connected to real accounts or files should use permission boundaries, audit logging, confirmation gates for destructive actions, and application-specific safety policies.

A Evaluation Details

ASIL is evaluated across diverse applications and task regimes using shared checkpoint-based validation, with GUI comparisons and audits of realization patterns. The benchmark emphasizes ASIL’s short trajectories while documenting residual access, data, and training-related failures.

  • Benchmark composition: 380 tasks span 15 applications, including 300 single-application and 80 multi-application tasks, with each single-application environment contributing 20 tasks.The default paired budget is 15 interaction steps, while complex GUI runs may use up to 50; ASIL trajectories average below five steps.
  • GUI comparison: ASIL’s GUI comparison includes a repaired runtime and permits up to 50 steps for complex tasks requiring additional recovery opportunities.The interface-effect case study retains a paired LibreOffice payroll run as a qualitative illustration.
  • GUI comparison: One LibreOffice payroll task passes under ASIL through one modify_file semantic action, whereas GUI control fails after 15 events in a repeated literal-tab paste loop.The task requires ten employee rows, unique IDs, non-negative salaries, and correct Net Pay formulas.
  • Failure analysis: Residual ASIL failures include GIMP access-path limits, Calc data-coverage gaps, and a Thunderbird SFT regression later recovered by RL.These cases are drawn from the held-out hard suite and summarized with aggregate pass counts from seven models.
  • Evaluation protocol: The evaluator scores typed checkpoints along ordered success paths and requires every checkpoint on a matched path for binary success.Checkpoints include spreadsheet values, file existence, document-text regexes, SVG structure, image statistics, and REST resource state.
  • Evaluation protocol: The realization audit distinguishes file-backed, native-scripting, and service/API pathways across the evaluated applications.The LibreOffice path-restriction study compares file-backed ODF and spreadsheet edits with script dispatch on the same 60 tasks.

B ASIL Adapter and Trace Implementation Details

ASIL implements one structured observation–semantic action protocol across file, scripting, and service pathways, with adapters, schemas, traces, and shared evaluation infrastructure. The same modality also supplies data for supervised fine-tuning and on-policy reinforcement learning.

  • Adapter architecture: ASIL adapters expose observe(), execute(action), and validate_action(action) while supporting optional synchronization, rendering, and launch hooks.These methods extract state, apply semantic actions, and check action validity for each application.
  • Adapter architecture: A normalized ASIL observation contains metadata, application state, interactive elements, environment context, navigation structure, and a textual data summary.Metadata records the observation source, while application state includes the current view, active document, and document path.
  • Realization pathways: ASIL realizes one protocol through file-backed, native-scripting, and service/API pathways that produce normalized trace artifacts.File-backed applications include LibreOffice and Draw.io; native scripting covers Blender and GIMP; service-backed examples include Gitea and Thunderbird.
  • Action and trace representation: Actions encode an action_type, target, and params object, with types including set_value, invoke_function, modify_file, api_call, navigate, and batch.Per-environment schemas specify valid action formats, parameters, examples, completion, and software-specific guidance.
  • Training data: SFT-v0 replays verified ASIL actions, while agentic-guided-v2 records verified GPT-5.4 trajectories for thought–action supervision.The v0 action is predetermined rather than model-generated, and retained rows have valid JSON with zero execution or annotation errors.
  • Training data: The selected SFT checkpoints are chosen by downstream ASIL-380 benchmark performance, including an intermediate epoch for the 9B run.The 2B run continues from a v0-pretrained checkpoint, while the 9B run trains on the deduplicated v0+v2 split.
  • Reinforcement learning: RL uses an on-policy GRPO-style curriculum with ASIL environments, external rollout services, and separate training and validation tasks.The final settings include a two-stage 9B schedule and a short single-stage 2B schedule whose best checkpoint occurs early.

D Training Task Generation Overlap Audit

The RL curriculum is generated by rewriting structured task templates, auditing lexical and structural overlap, and selecting quota-complete pools. The authors characterize the resulting source pool as engineering-valid but not fully de-templateized.

  • Generation pipeline: The generator creates structured task specifications by changing workplace scenarios, literal slots, artifacts, labels, paths, and other task values without exposing held-out final-test instructions.Deterministic replacement converts specifications into executable ASIL task JSON, followed by parser, evaluator, and quota audits.
  • Overlap audit: Overlap fingerprints combine lexical, character n-gram, structural, semantic, evaluator-rule, artifact, asset, and source features.The combined weights are 0.36 lexical, 0.18 character n-gram, 0.28 structural, and 0.18 semantic similarity.
  • Selected curriculum: The final v3 pool contains 512 training tasks and 128 validation tasks across all 15 software domains, with zero rejected or hard-rejected selected tasks.It uses quotas of 32 training and 8 validation tasks per non-GIMP application, doubled for GIMP.
  • Selected curriculum: The selected v3 training set has a 17.8% strict-accept ratio, and the authors describe it as engineering-valid rather than fully de-templateized.Many selected tasks remain in the warning band, and final RL uses a 320-task training subset with 80 validation tasks.
  • Overlap audit: Maximum overlap risk below 0.55 is accepted, [0.55, 0.72) is warned, and hard rejects or risk ≥0.72 are rejected.The audit reports candidate-generation iterations and selected task sets after quota selection and duplicate checks.

E Reproducibility

The authors release the ASIL inference and evaluation stack, together with code, models, benchmark resources, runtime images, and training data to support reproduction and extension.

  • Released resources: The release includes the ASIL inference and evaluation stack for reproducing the benchmark setup and extending it to new applications.Public resources are separated into repositories, checkpoints, benchmark tasks, runtime images, and training data.
  • Released resources: The public package covers adapters for all 15 applications, the shared observation–action protocol, evaluator implementations, and the semi-automatic ASILization pipeline.The release also provides the supporting benchmark and training artifacts described by the paper.
Loading 2608.26991v1…