PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-Callable Feature Coverage: Measuring Software Readiness for AI Agents

Zedong Peng

arXiv:2609.35789v1cs.SEcs.HC

TL;DR

The paper asks how software can support agents through structured, controllable channels rather than merely whether agents can operate GUIs. It introduces metrics for measuring agent-callable coverage and readiness, and finds readiness gaps across systems alongside strong task completion on accessible capabilities.

  • Problem

    GUI interaction can be fragile and outside software governance, while measuring agent readiness requires going beyond whether an API endpoint exists.

    Source: §I · p. 1

  • Method

    The paper defines ACFC and ARC to assess system coverage and capability-level Accessibility, Discoverability, and Controllability, then applies ARC across 30 systems and validates execution in a testbed.

    Source: §I · p. 2, A. RQ1: Agent Readiness Landscape · p. 6, §IX · p. 10

  • Results

    Across 30 systems and 1,299 capabilities, median ACFC was 0.55; API-only agents completed 56 of 58 accessible tasks, with documentation ablation providing evidence that Discoverability affects success.

    Source: §IX · p. 10

  • Takeaways & Limitations

    The study indicates that accessibility alone does not capture readiness: systems can have moderate API coverage while lagging in safety governance, and documentation matters for execution among accessible capabilities.

    Source: §IX · p. 10

    Capability definitions involve human judgment, mitigated through written coding guidance, recorded scoring evidence, and release of the per-capability mapping.

    Source: G. Threats to Validity · p. 10

Abstract

from arXiv · show

AI agents already operate graphical software through screenshot-based computer use, so the pressing question is not whether agents can operate software, but how well software supports them through structured, controllable channels. We formalize this need as the GUI-API parity principle: every capability available to human users through a graphical interface should also be accessible to agents through a structured, callable interface with appropriate safety metadata. To operationalize this principle, we introduce two contributions. First, Agent-Callable Feature Coverage (ACFC), a product-level readiness metric quantifying how much of a system's human-facing functionality agents can access and use. Second, Agent Readiness Conformance (ARC), a framework that scores each capability on three axes: Accessibility (can an agent call it), Discoverability (can an agent find and understand it), and Controllability (can an agent use it safely), each on a 0-3 scale, yielding a composite 0-9 readiness score. Drawing on the Richardson Maturity Model, ARC provides finer-grained assessment than binary coverage. In an empirical study of 30 software systems across five categories, ARC-based ACFC scores are substantially lower than Accessibility-only metrics would suggest: most systems achieve moderate API coverage but lag on agent-oriented documentation and safety governance. In a controlled testbed of ten deployed systems, API-only agents completed 56 of 58 tasks whose target capabilities were exposed through structured interfaces; a documentation ablation reduced this to 50 of 58, showing that Discoverability affects execution even among accessible capabilities, while inaccessible capabilities, included as negative controls, were not solved. These findings hold across three agent models, including Google's open-weights Gemma4 run locally.

I. INTRODUCTION

As agents increasingly execute software tasks, GUI-only access leaves them relying on fragile, ungoverned screen interaction, while structured interfaces may cover only part of a product’s human-facing functionality. The paper proposes the GUI–API parity principle and metrics for measuring agent access and readiness, then studies readiness across software systems and tests execution in a controlled testbed.

I. INTRODUCTION

  • GUI-only interaction is fragile and operates outside software permission, audit, and error-handling mechanisms, while APIs may expose only a subset of GUI capabilities.The paper calls the mismatch between human-facing and agent-callable capabilities a feature parity gap.Source: §I · p. 1
  • The GUI–API parity principle calls for human-facing capabilities to be available through structured agent interfaces with appropriate safety metadata.The paper argues that agents will interact with software regardless, so the issue is whether access uses governed channels or brittle screen simulation.Source: §I · pp. 1–2
  • ACFC measures how fully human-facing capabilities are exposed through official, stable, structured interfaces, while ARC scores Accessibility, Discoverability, and Controllability from 0–3 each.ARC’s three dimensions yield a composite readiness score from 0–9, extending binary covered-or-uncovered assessment.Source: §I · p. 2
  • The empirical study examines 30 systems across five categories, measuring readiness distributions, developing a feature-parity-gap taxonomy, and testing accessibility and discoverability in a controlled testbed.The testbed research question asks whether Accessibility is a substrate for execution and whether Discoverability affects success among accessible capabilities.Source: §I · p. 2
  • The paper’s contributions include defining CC and ACFC, introducing ARC, mapping capability-to-interface coverage across 30 systems, and behaviorally validating API-only task execution and documentation effects.The documentation ablation provides evidence that reduced Discoverability lowers success when Accessibility is held fixed.Source: §I · p. 2

II. BACKGROUND

The background frames software capabilities as outcomes mediated through interfaces, then distinguishes interaction modalities and defines capabilities at the user-goal level.

A. Requirements, Specifications, and Shared Phenomena

  • ACFC treats GUI actions, REST calls, and MCP invocations as shared phenomena through which different users interact with the same underlying capability.The framework measures whether the agent-facing projection covers the capability set exposed to human users.Source: A. Requirements, Specifications, and… · p. 2

B. Agent Interaction Modalities

  • Structured APIs provide stable, typed interfaces, while GUI automation is fragile and error-prone; hybrid approaches combine GUI and API interaction per task.The modality choice motivates measuring which capabilities are callable through structured interfaces rather than GUI-only access.Source: B. Agent Interaction Modalities · p. 2

A. Human-Facing Capability

  • A human-facing capability is a coherent, independently valuable user-goal outcome, not an individual interface action such as clicking a button.Examples include exporting a video as MP4 and viewing an analytics dashboard.Source: A. Human-Facing Capability · p. 2

B. Agent-Callable Counterpart

The section defines agent-callable counterparts as official, stable, structured interfaces rather than GUI simulation, then develops ARC and coverage metrics that distinguish access from broader agent readiness.

B. Agent-Callable Counterpart

  • An agent-callable counterpart is an official, stable, structured interface for invoking an equivalent or sufficient capability without GUI simulation.Unofficial, undocumented, or unstable endpoints do not qualify.Source: B. Agent-Callable Counterpart · p. 3

C. The Agent Readiness Conformance (ARC)

  • ARC scores each capability 0–3 on Accessibility, Discoverability, and Controllability, totaling a diagnostic 0–9 score across staged call, understand, and safely delegate requirements.Discoverability is an epistemic prerequisite for effective Controllability, and the additive score is not a claim that dimensions are substitutable.Source: C. The Agent Readiness Conformance (ARC) · p. 3

D. Coverage and Readiness Metrics

  • Callable Coverage measures the fraction of capabilities with A ≥2, while ACFC ranges from 0 to 1 and reflects quality across all three ARC dimensions.A system with high Callable Coverage but poor Discoverability or Controllability has lower ACFC than its Callable Coverage alone suggests; the empirical study scores capabilities through expert coding with API or documentation evidence.Source: D. Coverage and Readiness Metrics · p. 3

E. Feature Parity Gap

A feature parity gap occurs when a human-facing capability falls short of maximum on any ARC dimension; the paper identifies five empirically refinable gap types.

E. Feature Parity Gap

  • The taxonomy distinguishes missing projection, partial projection, granularity mismatch, semantic drift, and verification gaps in agent-callable counterparts.These cover absent APIs, incomplete functionality, differing abstraction levels or models, and insufficient evidence to confirm outcomes.Source: E. Feature Parity Gap · p. 3

A. Coverage Metrics and GUI-to-API Traceability

Prior coverage and agent-interaction research measure tests, APIs, events, or agent trajectories, whereas ACFC measures whether human-facing capabilities are structurally exposed to agents. ARC addresses agent-specific readiness dimensions, and the study samples 30 systems across five categories.

A. Coverage Metrics and GUI-to-API Traceability

B. Agent GUI, API, and Hybrid Interaction

  • Earlier agent studies compare API, hybrid, and GUI interaction or evaluate agents using structured tools; ACFC instead measures software capability exposure.The cited work includes API-based and hybrid agents, COM abstractions, and callable APIs for agents.Source: B. Agent GUI, API, and Hybrid… · p. 4

C. API Maturity Models

  • ARC responds to the lack of an academic framework for per-capability agent readiness across Accessibility, Discoverability, and Controllability, scoring each dimension from 0–3.Its rubric sums the three scores into a composite from 0–9.Source: C. API Maturity Models · p. 4, Table I · p. 4

D. Agent-Ready APIs and Tool Quality

  • Prior work finds that structurally available APIs can still be difficult for agents to use, motivating ARC’s inclusion of Discoverability and Controllability alongside Accessibility.Documentation quality is linked in prior work to planning failures, agent behavior, and test-generation quality.Source: D. Agent-Ready APIs and Tool Quality · p. 4

A. Software Sample Selection

  • The sample contains 30 systems across five categories, with six systems in each category.Categories include creative, productivity, developer, business, and open-source self-hosted tools.Source: A. Software Sample Selection · p. 4
  • Sampled systems required public feature and API documentation and sufficient maturity and user base to represent their category.The systems, their primary agent-callable interfaces, and ARC scores are listed together in Table II.Source: A. Software Sample Selection · pp. 4–5

B. Capability Extraction Pipeline

The capability extraction pipeline combines LLM-assisted candidate generation with expert review and goal-level granularity calibration. It targets 30–50 validated capabilities per system.

Candidate extraction

  • The pipeline uses an LLM to extract candidate capabilities from official feature pages, help documentation, product tours, and changelogs.Candidates must be goal-directed, user-visible, and independently valuable, rather than UI operations or technical endpoints.Source: B. Capability Extraction Pipeline · p. 5

Human validation and granularity

  • An expert reviewer merges, splits, or removes candidates and calibrates each capability as an independent user goal rather than an interface action or internal operation.The target is 30–50 validated capabilities per system; complex workflows are decomposed when they contain independently valuable sub-goals.Source: B. Capability Extraction Pipeline · p. 5
  • Every automatically extracted candidate is manually checked against the guideline, and the validated capability lists are released for inspection.Source: B. Capability Extraction Pipeline · p. 5

C. ARC Scoring Protocol

The protocol scores capabilities using official structured interfaces and a three-dimension rubric, with documented evidence for each score. Its controlled testbed uses state-verified tasks to examine execution across capability coverage levels and documentation conditions.

C. ARC Scoring Protocol

  • Each validated capability is assessed in official API, MCP, SDK, and CLI documentation across Accessibility, Discoverability, and Controllability.Accessibility measures interface coverage, Discoverability documentation quality, and Controllability safety and governance metadata; when Accessibility is zero, the other scores are zero by definition.Source: C. ARC Scoring Protocol · p. 5
  • Scoring excludes GUI automation, unofficial or undocumented endpoints, deprecated endpoints, and third-party wrappers without vendor endorsement.Source: C. ARC Scoring Protocol · p. 5

D. Coding Reliability

  • All 1,299 capabilities have expert-coded scores with cited endpoint or documentation evidence, enabling readers to inspect, challenge, or rescore the mappings.The authors rely on transparency and traceability rather than an inter-rater statistic.Source: D. Coding Reliability · p. 5

E. Granularity

  • ACFC depends on extraction granularity; the study defines capabilities at user-goal level and targets 30–50 per system, but leaves quantitative granularity sensitivity analysis to future work.Source: E. Granularity · p. 5

F. Agent Task Validation (RQ3)

  • The testbed covers ten locally deployed systems and 78 tasks spanning A0 through A2–A3 capabilities, with three agent models given natural-language goals and API references.The agents act through a logged HTTP tool; the models are Opus 4.8, Sonnet 4.6, and locally served Gemma 4 (31B).Source: F. Agent Task Validation (RQ3) · p. 5
  • Task success is verified against the target environment state using API read-back and a database query, with oracle checks limited to attributes named in the task goal.For A=0 tasks, the database query checks whether the target state was reached through any available API path.Source: F. Agent Task Validation (RQ3) · p. 5
  • The test compares full and minimal documentation for covered tasks using McNemar’s exact test, while A=0 negative controls are reported descriptively rather than as a stochastic comparison.A secondary, deliberately underpowered analysis reports the Spearman correlation between per-system Callable Coverage and success rate for ten systems.Source: F. Agent Task Validation (RQ3) · p. 6

VI. RESULTS

Across 30 systems, ARC-based ACFC reveals substantial readiness gaps beyond API availability, with Controllability the main bottleneck. The gap taxonomy shows that accessible capabilities commonly lack agent-oriented discoverability or safety governance.

A. RQ1: Agent Readiness Landscape

B. RQ2: Gap Taxonomy

  • Accessibility gaps affected 370 capabilities (28.5%): 156 had no projection, concentrated in creative and GUI-heavy productivity features, and 214 had partial projections.Source: B. RQ2: Gap Taxonomy · p. 7
  • Among accessible capabilities, 275 (24.1%) scored D0–D1, 721 (63.1%) were at the D2 ceiling, and 147 (12.9%) achieved D3.Source: B. RQ2: Gap Taxonomy · p. 7
  • Among accessible capabilities, 490 (42.9%) scored C0–C1, while only 134 (11.7%) achieved C3 governance; the C1 ceiling was the most common gap pattern.Source: B. RQ2: Gap Taxonomy · pp. 7–8
  • Three common profiles are A2/D2/C1 (about 35% of A≥2 capabilities), A2/D1/C0, and A1/D2/C1, spanning governance, documentation, and coverage gaps.Source: B. RQ2: Gap Taxonomy · p. 8

A. RQ3: Behavioral Validation of ARC Dimensions

A controlled testbed examined whether structured Accessibility supports API-only execution and whether Discoverability matters when capabilities are already callable. Success was high for accessible tasks, fell with stripped documentation, and was absent for negative controls.

A. RQ3: Behavioral Validation of ARC Dimensions

  • 56 of 58 accessible tasks were completed under the full API reference, with success checked against environment state.The test covered ten locally deployed open-source systems; the task oracle did not rely on agent self-report.Source: A. RQ3: Behavioral Validation of ARC… · p. 8, Table IV · p. 8
  • 56/58 covered-task completions fell to 50/58 with a minimal reference; six tasks regressed and none improved (McNemar’s exact test p = 0.031).Regressions concentrated in tasks with non-obvious request payloads; conventional REST-body operations were unaffected.Source: A. RQ3: Behavioral Validation of ARC… · p. 8
  • 0 of 19 tasks with no official callable counterpart were completed in the API-only setting, serving as negative controls rather than evidence that ACFC predicts performance.The agent was instructed to report inability when no documented endpoint applied.Source: A. RQ3: Behavioral Validation of ARC… · p. 8
  • The per-system Callable Coverage correlation with success was positive but nonsignificant (ρ = 0.38, n = 10) in a deliberately underpowered check.The estimate was attenuated by oversampled A = 0 tasks and two systems contributing only accessible tasks.Source: A. RQ3: Behavioral Validation of ARC… · p. 9
  • Accessibility was a necessary substrate, not a sufficient condition: the two covered failures reflected task decomposition and API payload ergonomics.One task omitted required sub-attributes; another required a complete-object PUT payload.Source: A. RQ3: Behavioral Validation of ARC… · p. 9
  • Covered-task completion was 56/58 for Opus 4.8, 55/58 for Sonnet 4.6, and 45/58 for Gemma 4; all three models left the 19 negative controls unsolved.The two hardest covered tasks failed for all three models.Source: A. RQ3: Behavioral Validation of ARC… · p. 9

VIII. DISCUSSION

The discussion interprets observed feature gaps as a software-engineering problem and outlines different remediation paths across software categories. It also connects agent accessibility to broader accessibility and governance concerns, while noting the risks of under-governed callable interfaces and the labor involved in manual ARC scoring.

A. Implications for Software Engineering

  • Creative tools face a challenge in modeling spatial and temporal operations, while productivity and business tools can more tractably extend APIs to cover administrative, analytical, and approval-flow capabilities.Across categories, adding risk annotations, operation classifications, and human-in-the-loop governance to already-accessible operations is presented as a natural first step.Source: A. Implications for Software Engineering · p. 9

B. Toward AI-Accessible Software

  • Agent-accessible APIs and metadata can also serve CI/CD, testing, and third-party integrations, while Controllability investments can address broader compliance needs.The discussion maps ARC dimensions to WCAG principles and proposes graduated conformance targets.Source: B. Toward AI-Accessible Software · p. 9

C. Two Failure Modes of GUI–API Parity

  • Callable, discoverable APIs can expose operations without operation-level risk classification, confirmation, or audit metadata; the paper reports this as a structural risk, not a measured vulnerability rate.The agents held authorized administrator credentials, and a dedicated security analysis is outside the paper’s scope.Source: C. Two Failure Modes of GUI–API Parity · pp. 9–10

D. Toward Automated ARC Assessment

  • Manual ARC scoring required 3,900 individual scores across 30 systems; the authors describe semi-automated assessment as feasible using OpenAPI specifications.Proposed checks cover endpoint existence, description richness, and authentication scopes.Source: D. Toward Automated ARC Assessment · p. 10

E. Implications for Agent Framework Design

The paper concludes that API coverage alone misses readiness gaps, especially in Controllability, and reports behavioral evidence that both Accessibility and Discoverability matter for API-only execution. It also qualifies its scoring and validation: the additive formula is retained for the observed score pattern, while sample scope and judgment-based capability definitions constrain interpretation.

E. Implications for Agent Framework Design

  • Agent frameworks should account for incomplete API coverage, assess documentation before tool use, and plan conservatively for endpoints with low Controllability.The discussion recommends human fallback when full equivalence is unavailable.Source: E. Implications for Agent Framework… · p. 10

F. Staged vs. Additive Scoring

  • The additive ARC score is retained because observed mean scores followed A ≥ D ≥ C, making the practical difference from the staged alternative small in this dataset.The staged formulation remains a subject for future investigation.Source: F. Staged vs. Additive Scoring · p. 10

G. Threats to Validity

  • Capability definitions involve human judgment, covered-versus-uncovered results may be confounded, the system sample may not represent all software, and LLM-assisted extraction may be inconsistent.The authors mitigate these threats with written guidelines, multiple models, state-based oracles, manual validation, and released evidence; the RQ3 validation is limited to self-hosted open-source systems.Source: G. Threats to Validity · p. 10

IX. CONCLUSION

  • Across 30 systems and 1,299 capabilities, median ACFC was 0.55, and Controllability was the primary readiness bottleneck; validation found 56 of 58 accessible tasks completed and fewer completions with stripped documentation.The 30-system sample showed median ACFC of 0.69 for developer tools and 0.41 for creative tools; overall mean C was 1.42.Source: §IX · p. 10

Related papers

Loading 2609.35789v1…