PaperlayerView exact revision on arXiv

Source-linked AI summary

A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution

Alan Wu, Wei Tao, Yiwen Guo

arXiv:2610.15765v1cs.SEcs.AI

TL;DR

Desktop applications expose interfaces designed for humans and developers, leaving unclear which interface properties best support agents. This paper uses software evolution to study those properties and synthesizes agent-native interfaces; across the expanded benchmark, atomic commands outperform direct SDK use, with further gains from goal-level commands.

  • Problem

    Desktop interfaces require agents to interpret visual controls or discover and compose low-level operations, and the interface properties that make applications easier for agents to use remain unclear.

    Source: §1 · p. 1

  • Method

    The paper varies software releases in a controlled benchmark to identify interface properties, then develops ANIS to synthesize runtime-grounded commands and verified goal-level compositions.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    72.3% success with atomic commands versus 68.5% with direct SDK use and 17.4% with CLI-Anything; goal-level commands raise success to 74.5%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across eight applications, ANIS raises success from 68.5% with the native SDK to 74.5%, while removing observation commands costs 12.0 points.

    Source: §6 · p. 11

    Implicit object relationships in FreeCAD and Blender make some operations difficult to validate automatically, leaving a small set of native-SDK capabilities uncovered.

    Source: §5.3.2 · p. 10

Abstract

from arXiv · show

Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hinder even capable agents, yet it remains unclear what interface properties make desktop software easier for agents to use. To study this question, we treat software evolution as a natural source of interface variation and introduce DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six desktop applications. By holding the task, agent, and verifier fixed while varying the software release, we measure how interface changes affect agent performance. Across two models and six applications, the newest release consistently outperforms the oldest. Trajectory analysis identifies three relevant interface properties: discoverability, granularity, and observability. Guided by these properties, we develop ANIS, a framework that synthesizes agent-native interfaces from the capabilities of installed applications. On an expanded evaluation of 368 tasks across eight applications, atomic interfaces improve success over native SDK use by 3.8 and 3.5 percentage points with DeepSeek V4 Flash and Claude Haiku 4.5, respectively, while reducing online token usage by approximately 73%. Goal-level commands further raise success to 74.5% and 67.4%, respectively.

1 Introduction

Desktop interfaces built for people and developers leave agents to interpret visual controls or assemble low-level operations, and the interface properties that help agents remain unclear. This paper studies interface variation across software releases, identifies three relevant properties, and introduces ANIS, which improves task success on an expanded benchmark.

1 Introduction

  • Holding the agent, tasks, and verifier fixed while varying software releases uses software evolution as a controlled source of interface variation.The benchmark contains 255 artifact-verified tasks across six applications and five domains.Source: §1 · p. 2
  • The newest release beats the oldest in all 12 model–application series; QGIS success rises from 33% to 65% as mean SDK operations per task fall from 30.4 to 21.2.Impress gains 19 points, from 33% to 52%.Source: §1 · p. 2
  • Trajectory analysis identifies discoverability, granularity, and observability as interface properties relevant to agent performance.Discoverability concerns finding and understanding operations, granularity concerns goal-aligned operations, and observability concerns reading back modified artifact state.Source: §1 · p. 2
  • 72.3% success with atomic commands versus 68.5% with direct SDK use and 17.4% with CLI-Anything; goal-level commands raise success to 74.5%.Results are on the expanded benchmark.Source: §1 · p. 2
  • ANIS synthesizes runtime-grounded atomic commands and verified goal-level compositions, pairing writable properties with read commands for artifact verification.Goal-level commands are composed from atomic commands and released after replay reproduces the expected saved state.Source: §1 · pp. 2–3

2 Related Work

Prior desktop-agent work improves interaction through GUI/API combinations, abstractions, and discovery tools, while benchmarks often vary the agent with the interface fixed. This paper instead varies the interface while holding task, agent, and verifier fixed.

2 Related Work

  • Prior benchmarks and systems commonly hold the interface fixed while changing the agent; this work fixes the task, agent, and verifier to study interface design.The cited systems include execution-checked desktop benchmarks and screen-based interaction agents.Source: §2 · p. 3
  • Unlike MCP-oriented work focused on discovering and invoking existing operations, this work synthesizes verified atomic operations, state-read commands, actionable errors, and validated goal-level sequences.The atomic result can be exposed through CLI or MCP, while publishing goal-level sequences remains future work.Source: §2 · p. 3

3 Benchmark Construction

DeskIF-Bench supports controlled cross-version evaluation through artifact-level tasks, deterministic verification, and diverse applications. Its construction combines reused and synthesized tasks, with feasibility and verifier checks across releases.

3 Benchmark Construction

  • DeskIF-Bench uses deterministic artifact-verification scripts, interface-independent outcomes, and coverage of at least 30 tasks per application.The benchmark spans six applications across five domains.Source: §3 · p. 3

3.1 Software Selection

  • The six selected applications span five domains and varied artifact types, from office documents and vector graphics to 3D scenes, geospatial data, and parametric CAD models.Applications are Writer, Impress, Inkscape, Blender, QGIS, and FreeCAD.Source: §3.1 · p. 4

3.2 Task Construction

  • Tasks specify artifact outcomes, are checked by dedicated deterministic verifiers, and are screened for feasibility in every evaluated release.Verifiers use structured artifact inspection or pixel-level comparison under fixed tolerances, and are checked against reference artifacts and task-relevant perturbations.Source: §3.2 · pp. 4–5
  • The benchmark contains 255 tasks: 102 retained from prior datasets and 153 synthesized to fill coverage gaps.Synthesized tasks use an authoring agent to produce an instruction, reference artifact, and executable verifier; authoring skills or privileged libraries create capability asymmetry with the solving agent.Source: §3.2 · pp. 4–5

4 Empirical Study

The empirical study varies application releases while keeping evaluation conditions fixed, then traces performance differences to interface properties. Newer releases consistently outperform older ones, while observability remains limited across evaluated applications.

4.1 Experimental Setup

  • The comparison holds the task, seed files, scaffold, permissions, time budget, and verifier fixed while varying only the application release.Success is reported as pass@3; trajectory-level comparisons are restricted to model–task pairs verified successful in all four releases.Source: §4.1 · p. 5

4.2 Performance Across Software Versions

  • The newest release outperforms the oldest in all 12 model–application series across two backbones.For QGIS, DeepSeek rises from 41% to 69% and Haiku from 33% to 65%; Impress rises from 40% to 58% and from 33% to 52%, respectively.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3.1 Properties Revealed by Cross-Version Comparison

  • Greater cohesion and richer operation descriptions improve discoverability; QGIS runtime descriptions grow by 20% per function from version 3.10 to 3.44.FreeCAD 1.0 makes reusable model parameters explicit through App::VarSet, whereas FreeCAD 0.21 encoded them indirectly as aliased spreadsheet cells.Source: §4.3.1 · p. 6
  • Goal-aligned operations reduce low-level composition: Haiku’s mean QGIS chain length falls from 30.4 to 21.2 operations.QGIS’s Processing toolbox grows from 272 to 440 operations; Inkscape’s multi-page export task falls from 101 turns and 1.8M input tokens to 30 turns and 119k.Source: §4.3.1 · p. 6, Table 1 · p. 7

4.3.2 Property Limited across Releases

  • 581 trajectories reported completion but failed artifact verification; 569 contained a detected check, and 416 of those had independent artifact readback.A further 153 of the 569 checked trajectories (26.9%) lacked independent readback; these counts do not attribute failures to agent reasoning or unavailable state.Source: §4.3.2 · p. 7
  • Some QGIS properties are easy to modify but hard to inspect at the same abstraction level, including vertex-marker appearance, which has no corresponding reader.The authors conclude that discoverability and granularity improve in several newer releases, while observability remains limited across applications.Source: §4.3.2 · p. 7

5 Synthesizing Agent-Native Interfaces

The section presents ANIS, which synthesizes agent-oriented interfaces from installed applications, then evaluates atomic and goal-level commands against native SDKs and CLI-Anything. Its two-phase design targets discoverability, granularity, and observability.

5.1 Atomic Commands

  • ANIS discovers capabilities from runtime operations, introspected objects and services, and application menus, then tests candidates to retain persistent artifact changes or state inspection.Discovery prioritizes recall, while command synthesis prioritizes precision; transient application state and runtime machinery are filtered out.Source: §5.1 · p. 8
  • Each admitted capability becomes a documented read, write, or object-operation command, validated by saving and reopening the artifact to check that its effect persists.ANIS groups commands by artifact object and action, and exposes read commands when a corresponding accessor exists for a writable property.Source: §5.1 · p. 8

5.2 Goal-Level Commands

  • Recurring command sequences become goal-level commands only when they realize a shared artifact-level subgoal, and each released command passes a reproducibility check on a clean artifact copy.ANIS surfaces sequences recurring in at least three requests for review; recurrence alone does not establish a shared goal.Source: §5.2 · p. 9

5.3.1 Experiment Settings

  • The evaluation uses 368 tasks across eight applications, including Calc and GIMP tasks held out from method development, and compares ANIS with native SDK and CLI-Anything interfaces.Calc and GIMP interfaces were synthesized after the pipeline was frozen and were not inspected before evaluation.Source: §5.3.1 · p. 9

5.3.2 Experiment Results

  • 72.3% success with atomic commands versus 68.5% with the native SDK, rising to 74.5% with goal-level commands; tokens per task fall from 84.4K to 22.6K and 21.2K, respectively.Adding goal-level commands never lowers success on any application; atomic and augmented interfaces remain below the native SDK on FreeCAD and Blender.Source: Table 2 · p. 10, §5.3.2 · p. 10
  • ANIS exposes over 20× as many commands as CLI-Anything across shared applications, and 98.7% of CLI-Anything failures stem from missing commands rather than agent errors.The authors attribute CLI-Anything’s 17.4% success rate to omitted capabilities, not to the command-line form itself.Source: §5.3.2 · p. 10
  • Implicit preconditions leave some native-SDK capabilities uncovered in FreeCAD and Blender, where operations depend on linked objects, selections, or editing modes.28.6% of ANIS’s FreeCAD failures lack a required command; this gap occurs only in the two applications with the most implicit object relationships.Source: §5.3.2 · p. 10
  • Prohibiting observation commands reduces overall success from 72.3% to 60.3%, a drop of 12.0 percentage points.The ablation leaves editing commands available while prohibiting observation commands through agent instructions.Source: §5.3.2 · p. 10

6 Conclusion

The conclusion presents ANIS as a way to synthesize agent-oriented interfaces without waiting for application evolution. The paper also describes its use of generative AI and provides its code, tasks, verifiers, and evaluation scripts.

6 Conclusion

  • Across eight applications, ANIS raises success from 68.5% with the native SDK to 74.5%, while removing observation commands alone costs 12.0 points.ANIS combines runtime capability commands with goal-level commands mined from use.Source: §6 · p. 11

AI Use Statement

  • The authors used LLMs to polish text, find candidate related work, implement experiment and analysis code, and synthesize benchmark tasks, with human review and verification.They report checking edits, reviewing and testing AI-assisted code, checking references, and accepting tasks only after verifier checks.Source: AI Use Statement · p. 11

Reproducibility Statement

A Overall Model Performance and Documentation

The appendix compares native-SDK task success with and without official documentation across four backbones and eight applications. Documentation does not consistently improve success and can increase exploration overhead.

A Overall Model Performance and Documentation

  • The comparison evaluates four backbones through the OpenCode scaffold on the same tasks and artifact verifiers, with and without official documentation.The documentation condition provides offline copies and instructs agents to consult them when uncertain.Source: A Overall Model Performance and… · p. 14
  • Documentation raises Kimi’s success by 1.4 percentage points, while Haiku, DeepSeek, and Opus decline by 1.1, 1.7, and 1.9 points, respectively.Effects also vary within a model: Haiku improves on GIMP but declines on Blender.Source: A Overall Model Performance and… · p. 14

B Token Budget Distribution

The figure compares how two models allocate tokens across exploration, authoring, and verification/repair, with and without additional documentation. Requiring consultation of the documentation increases exploration, but does not consistently improve success.

B Token Budget Distribution

  • 36.2% vs 28.8% for Haiku and 32.5% vs 24.1% for Kimi are the exploration-token shares with versus without the added documentation library.Adding the library and requiring consultation increases the exploration share for both models.Source: B Token Budget Distribution · p. 14
  • Kimi devotes more tokens to writing and executing code, while Haiku distributes its budget more evenly between implementation and checking or repair.Source: B Token Budget Distribution · p. 14
  • Figure 4 compares token allocation across exploration, authoring, and verification/repair for two models with bundled references alone versus an additional documentation library.Exploration includes consulting references, probing interfaces, and inspecting task files; authoring includes planning, writing, and executing code.Source: B Token Budget Distribution · p. 14, Figure 4 · p. 15

C Performance Evaluation with Claude Haiku 4.5

With the same scaffold, 368 tasks, and generated interfaces, Claude Haiku 4.5 shows the same pattern as DeepSeek: ANIS improves success, with goal-level commands performing best.

C Performance Evaluation with Claude Haiku 4.5

D Observation Command Ablation

Prohibiting observation commands reduces task success, while extending interfaces with goal-level commands improves results where prior task trajectories reveal recurring subgoals.

D Observation Command Ablation

  • Because software evolution did not deliver observability, the ablation prohibits observation commands while keeping the other experimental settings fixed.Source: D Observation Command Ablation · p. 15
  • Prohibiting observation reduces overall success from 72.3% to 60.3%, a drop of 12.0 percentage points; access to editing commands alone is often insufficient without artifact state to choose parameters and detect deviations.Source: D Observation Command Ablation · p. 16
  • Calc and Writer fall by 30.3 and 19.7 points, while GIMP, Impress, and Inkscape show no aggregate change.In a Blender task, reading the chair’s initial orientation and checking the saved scene succeeds; a 180° relative rotation without observation leaves it perpendicular and fails verification.Source: D Observation Command Ablation · p. 16

E.1 Full-Task Evaluation

  • Blender improves by 5.7 points, Calc by 5.2, and QGIS by 3.7 with goal-level commands; the five other applications are unchanged.No goal-level command was promoted for those five applications because few subsequences recurred across their requests; on tasks invoking a goal-level command, input tokens fall by 6.2%.Source: §E.1 · p. 16

E.2 Held-Out Evaluation

On held-out tasks, goal-level commands improve success and reduce token use. The paper also reports interface-generation time, evaluated software releases, and task-count categories.

E.2 Held-Out Evaluation

  • The held-out test contains 110 tasks from a per-application split of 368 tasks into 258 training and 110 test tasks; synthesis retains seven commands for Calc, QGIS, and Blender.Only training-task trajectories are used to synthesize the commands.Source: §E.2 · p. 16
  • 2.7 percentage points: adding goal-level commands raises held-out successful tasks from 64 to 67 out of 110 and reduces tokens per task from 22.8K to 21.7K (4.8%).Source: Table 6 · p. 17, §E.2 · p. 17

F Interface Generation Time

G Evaluated Software Versions

  • The cross-version experiments use each application’s newest available release and earlier releases spaced at approximately uniform intervals, with the installed release as the only manipulated factor.The LibreOffice release row applies to Writer and Impress.Source: G Evaluated Software Versions · p. 17, Table 8 · p. 17

H Task Counts by Application

  • Task counts are reported per application, distinguishing reused tasks from tasks synthesized through the authors’ pipeline.Source: Table 9 · p. 18

I Reusing Existing Benchmarks

The section describes filtering benchmark tasks for persistent, artifact-verifiable outcomes, then organizing commands around artifact objects and actions, from atomic operations to goal-level sequences.

I Reusing Existing Benchmarks

  • Tasks are retained only when deterministic artifact-based verification can inspect the resulting artifact, excluding presentation-only and transient interface-state tasks.The source benchmarks provide realistic software-use tasks with executable evaluators.Source: I Reusing Existing Benchmarks · p. 18

J.1 Atomic Command Organization

  • ANIS groups commands by the artifact objects they operate on and names them by object and action, letting agents browse or search before inspecting relevant specifications.For QGIS, commands for layouts and layers are grouped under their respective objects; this keeps the interface comprehensive without placing the full command set in context.Source: §J.1 · p. 18

J.2 Goal-Level Command Construction

  • Goal-level commands package ordered atomic steps for a shared artifact-level subgoal, treating request-specific targets or option values as inputs and stopping with an error if a step fails.The model derives the command from atomic-command descriptions and originating requests; shared steps and values remain fixed.Source: §J.2 · p. 18

Related papers

Loading 2610.15765v1…