Source-linked AI summary
Callability Is Not Operability: Controlled Interface Interventions for LLM Agents
Zihao Wang
TL;DR
Reliable autonomous operation requires interfaces to expose action-relevant state and semantics, not merely accept syntactically valid calls. The paper formalizes this gap through AFT and evaluates its mechanisms with AFT-Bench, finding consistent mechanism-specific improvements across controlled experiments.
Problem
A valid tool call may leave an agent unable to distinguish operational states that require different continuation actions, and additional reasoning cannot guarantee the correct decision.
Method
The paper defines AFT interface semantics for discovery, execution recovery, external effects, machine-readable results, and verification, then uses controlled AFT-Bench comparisons that vary only the interface treatment.
Results
Across controlled experiments, selective discovery reduces context exposure without sacrificing capability recall, recovery mechanisms address distinct failures, effect semantics reduce unsafe duplicate actions, and verification corrects claims that disagree with world state.
Takeaways & Limitations
Tool-interface semantics can improve operability through distinct mechanisms rather than reducing the evaluation to model capability differences.
Takeaways & Limitations
The experiments provide conditional evidence about interface mechanisms under activated failures, not estimates of failure prevalence or generalizable effect sizes across production backends.
Abstract
from arXiv · showhide
A tool call can be perfectly valid yet still leave an autonomous agent unable to determine what to do next. For example, if an external effect commits but its response is lost, committed and uncommitted states may become indistinguishable to the agent even though they require different continuation actions. We study this gap between callability and operability: whether a tool interface exposes the action-relevant state and semantics needed for an agent to continue safely under operational uncertainty. We operationalize tool operability through Agent-First Tooling (AFT), a set of interface mechanisms spanning selective capability discovery, execution lifecycle and recovery, explicit external-effect semantics, machine-readable results, and postcondition verification. We introduce AFT-Bench, a controlled interface-intervention framework that holds the task, backend, initial state, injected failure, agent, and language model fixed while varying the interface exposed to the agent.
1 Introduction
The paper distinguishes syntactic callability from agent operability: reliable continuation requires interfaces to expose action-relevant state and semantics under operational uncertainty. It introduces AFT and AFT-Bench to isolate mechanism-specific interface effects while holding task, backend, failures, agent, and model fixed.
- Problem: Operational ambiguity arises when different underlying states require different continuation actions but remain indistinguishable through the available interface.A committed effect with a lost response may invite an unsafe retry, while an uncommitted effect may require retrying.
- Evaluation: The empirical program combines frozen deterministic experiments, persistent SQLite replication, and adaptive agents from three LLM families across six workloads.The resulting matrix contains 18/18 completed model–workload cells and 2,385 result rows.
- Results: Selective discovery, recovery mechanisms, effect semantics, and verification produce large mechanism-specific effects, with recovery stable across models and verification benefits model-dependent.In pooled analysis, discovery reduces context exposure by approximately 4,013 tokens, recovery mechanisms improve matched recovery by 100 percentage points, effect semantics reduce duplicate and unsafe effects by 56.9 and 50.0 percentage points, and verification reduces incorrect terminal claims by 27.8 percentage points.
- Conclusion: Tool-use reliability depends jointly on model policy and the operational semantics exposed at the agent–tool boundary.Stronger reasoning can compensate for some ambiguity but cannot guarantee the correct continuation when action-relevant states remain observationally indistinguishable.
- AFT formulation: AFT defines agent operability through capability discovery, execution and recovery, external-effect semantics, structured results, and postcondition verification.These mechanisms expose action-relevant information across the lifecycle of tool use.
- Method: AFT-Bench varies interface semantics while holding the task, backend, initial state, injected failure, controller, and model fixed.Its paired comparisons, cumulative ladder, primitive ablations, fault injection, and state-based verification target mechanism-specific effects rather than a generic leaderboard.
2 Related Work
Related work evaluates tool-using agents and increasingly richer interfaces, but the paper positions operability as broader than context control, interaction overhead, or agent capability alone. Its interventions additionally target recovery, durable state, external-effect semantics, and postcondition verification under matched failures.
- Agent evaluation: Prior benchmarks establish that realistic tool use is stateful and difficult, while primarily evaluating agent capabilities within a given tool environment.Examples include BFCL, τ-bench, and ToolSandbox.
- Interface infrastructure: On-demand discovery and programmatic execution address capability-catalog context pressure and repeated model–tool round trips.These mechanisms are presented as infrastructure developments rather than complete operability solutions.
- Positioning: The paper extends richer interface semantics toward controlled evaluation of execution recovery, durable state, external effects, and postcondition verification.This scope goes beyond the context and interaction overhead addressed by deployment practices.
3 Agent-First Tooling Model
The Agent-First Tooling model treats operability as the interface’s ability to expose or enforce machine-actionable semantics for safe autonomous continuation under uncertainty. It represents this interface across capability, execution, effect, verification, and governance semantics, and evaluates distinct operational dimensions rather than a single score.
- System model: AFT models an agent acting through a tool on an external, stateful backend while recording execution, recovery, effects, verification, and backend-state changes.The model includes faults such as transport interruption, lost process state, response loss after commit, stale state, authorization changes, incorrect terminal reports, and partial completion.
- System model: The backend environment comprises states, operations, operation-induced transitions, and evidence obtainable from the backend or runtime.Operations may be read-only, reversible, compensatable, or irreversible.
- From callability to operability: Operational ambiguity occurs when distinct external histories yield the same interface observation even though they require different safe continuation actions.A stronger model may guess which state is more likely, but identical observations prevent a policy from guaranteeing the correct continuation in both histories.
- Mechanisms as ambiguity resolution: Interfaces can reduce ambiguity by exposing additional evidence or by making the same continuation safe across multiple hidden states.Verification primarily improves state distinction, while idempotency and guarded effect semantics primarily stabilize continuation; lifecycle and recovery mechanisms may do both.
- From callability to operability: Agent-operability requires machine-actionable semantics for capability discovery, execution tracking and recovery, safe external effects, and terminal-outcome verification.Operability is represented as a multidimensional profile because mechanisms address different failure modes and deployments may value dimensions differently.
- Operational interface semantics: The interface tuple groups capability and representation semantics, execution semantics, and effect and governance semantics above the transport boundary.The groups cover exposed operations and representations; invocation identity, lifecycle, continuation, cancellation, and recovery; and effects, idempotency, compensation, verification, authority, approval, and audit.
- Mechanisms as ambiguity resolution: Resume and durability address different recovery problems, while effect semantics and verification address different aspects of post-call uncertainty.Resume continues a known invocation after interruption; durable state reconstructs an invocation after process-local state loss; verification supplies evidence about the actual outcome.
- Mechanisms as ambiguity resolution: Structured output carries lifecycle, effect, recovery, and verification semantics but is not itself an independent safety guarantee.The shared criterion is whether the interface gives the agent and runtime a reliable way to choose a safe continuation when execution becomes uncertain.
4 Experimental Design
The study treats interface semantics as the experimental treatment, holding task, backend, state, faults, controller, and model fixed while varying only the relevant interface conditions. It combines deterministic mechanism identification, persistent SQLite replication, and adaptive evaluation across six workloads, while measuring conditional effects rather than production failure prevalence.
- Experimental controls: Paired comparisons vary only the interface treatment while matching the task, backend, initial state, fault schedule, controller, model, and provenance.Stronger interfaces may expose lifecycle, recovery, effect, or verification semantics but cannot add business capabilities or perform tasks for the agent.
- Evidence layers: Deterministic experiments first validate fault activation, treatment isolation, recovery paths, and safety labels before adaptive LLM evaluation.The deterministic evidence, task definitions, interface treatments, fault semantics, controller contract, and verifiers were frozen before LLM validation.
- Evidence layers: Key effect-safety findings are replicated on a persistent SQLite-backed environment to test duplicate effects, stale updates, and unsafe commits across durable transactions.This replication checks whether safety effects remain meaningful beyond an in-memory simulator.
- Adaptive evaluation: All 3 × 6 = 18 model–workload cells completed, yielding 2,385 result rows across three adaptive model families and six canonical workloads.The workloads cover discovery, interruption recovery, post-commit response loss, stale state and permission drift, terminal verification, and partial-success verification.
- Workloads and measures: Discovery experiments scale catalogs over |D| ∈ {10, 50, 200, 1000} while measuring context exposure, schema materialization, top-k recall, fallback behavior, and final capability selection.Recall is evaluated independently of task completion so token reduction is not conflated with successful discovery.
- Workloads and measures: Recovery and safety workloads separate transient interruption, process-local state loss, response loss after committed effects, stale state, permission drift, and incorrect terminal reports.The benchmark records acceptance, execution start, commit crossing, response delivery, and subsequent recovery or reconciliation; transport retransmission, continuation, re-execution, and reconciliation are distinct operations.
- Robustness analysis: Recovery effects are especially stable across models, whereas the marginal benefit of verification is more model-dependent in leave-one-model-out analyses.These analyses examine the three pairwise model subsets to assess whether pooled conclusions are driven by one model family.
5 Results
The results show distinct interface mechanisms addressing distinct operational failures: discovery reduces context exposure, recovery mechanisms restore interrupted work, effect semantics improve action safety, and verification improves terminal-claim accuracy. These effects are generally robust across models for runtime mechanisms but more model-dependent for policy-sensitive mechanisms.
- 5.1 Selective Discovery Compresses Context Without Sacrificing Recall: Selective discovery reduces tool-context exposure by 4013.17 tokens on average while preserving recall within the pre-specified non-inferiority margin.The paired recall difference is 0.0139, with a 95% confidence interval of [0.0000, 0.0417], above the −0.10 margin.
- 5.2 Recovery Depends on Matching Interface State to Failure Mode: Resumable invocation and durable execution state each improve recovery by 1.00 utility unit under their matched interruption and state-loss treatments.Both effects have task-clustered intervals of [1.00, 1.00], but the findings apply to the specific failure classes targeted by each mechanism.
- 5.6 Summary: Recovery effects remain invariant across the three evaluated model families, while effect safety and terminal verification vary more by model.The pooled effects remain positive, but interaction evidence is exploratory because only eight task clusters contribute to the relevant analysis.
- 5.4 Verification Corrects Incorrect Terminal Beliefs: Postcondition verification reduces incorrect terminal claims by 0.2778 pooled utility across 144 matched pairs.In a secondary partial-success workload, verification improves terminal-claim correctness by 0.5185 across 27 pooled pairs.
- 5.6 Summary: Interface mechanisms have conditional value: runtime distinctions remain necessary across models, whereas some policy-sensitive mechanisms overlap with behaviors stronger or more conservative models already implement.
- All seven pre-specified pooled comparisons satisfy their planned inferential criteria after multiplicity correction.
6 Limitations and Future Work
The evaluation prioritizes mechanism identification under controlled failures, limiting claims about real-world prevalence, backend generalization, model universality, production-scale performance, and automatically recoverable semantics.
- Effect sizes are conditional evidence for what interface mechanisms do when relevant failures occur, not estimates of deployed-system failure prevalence.
- Most experiments use resettable synthetic environments, while SQLite replication reduces simulator dependence without establishing production-backend generalization.
- Extending the controlled design to additional independently implemented systems remains an important direction for future work.
- Three contemporary LLM families reveal model–interface heterogeneity but cannot establish model-independent universality; recovery is stable, whereas verification value varies substantially.
- The study emphasizes reliability, recovery, context exposure, and effect safety rather than production-scale timing, queueing, distributed execution, or high-concurrency behavior.
- Legacy systems may require developer or policy input to specify idempotency, authority, compensation, and postcondition evidence, motivating safer contract derivation and governance.
7 Conclusion
The paper argues that valid syntax is insufficient for reliable autonomous tool use when execution or external state is uncertain. AFT-Bench isolates interface mechanisms and finds a consistent mechanism-specific pattern, supporting interfaces that preserve action-relevant distinctions for safe continuation.
- Stateful or effectful execution requires agents to determine what happened, which distinctions matter, and what continuation is safe.
- AFT defines interface semantics for discovery, lifecycle recovery, external effects, machine-readable results, and postcondition verification, while AFT-Bench isolates their effects.
- Across controlled experiments, a persistent backend, and three adaptive LLM families, interface mechanisms show a consistent, mechanism-specific pattern rather than a model-capability comparison.
- Reliable autonomous tool use depends on interfaces preserving or stabilizing action-relevant distinctions needed to choose the next safe action.