PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal Agents Suffice for Enterprise Automation

Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar

arXiv:2604.00073v3cs.SEcs.AIcs.CL

TL;DR

It remains unclear whether complex GUI-driven and tool-augmented agent architectures are necessary for enterprise automation. Across diverse enterprise benchmarks, minimal terminal agents using direct API interaction matched or outperformed them at lower cost.

  • Problem

    It remains unclear whether complex agentic abstractions are necessary for enterprise automation when stable APIs are available.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    The paper evaluates GUI-driven, MCP-based tool-augmented, and minimal terminal agents on identical enterprise benchmarks using the same LLM backbone.

    Source: §3 · p. 3

  • Results

    8 of 12 platform–model combinations show terminal agents matching or exceeding web-agent accuracy, while consistently costing less and outperforming tool-use agents in accuracy.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    Terminal-based, API-first agents are a strong default foundation for enterprise automation on platforms exposing expressive APIs.

    Source: §6 · p. 11, Limitations · p. 12

    The evaluation covers three production-grade platforms and 729 single-session tasks, excluding long-horizon, cross-platform, and human-oversight scenarios.

    Source: Limitations · p. 12

Abstract

from arXiv · show

There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously. Among the approaches explored are tool-augmented agents built on abstractions such as Model Context Protocol (MCP) and web agents that operate through graphical interfaces. Yet, it remains unclear whether such complex agentic systems are necessary given their cost and operational overhead. We argue that a coding agent equipped only with a terminal and a filesystem can solve many enterprise tasks more effectively by interacting directly with platform APIs. We evaluate this hypothesis across diverse real-world systems and show that these low-level terminal agents match or outperform more complex agent architectures at a fraction of the cost. Our findings suggest that simple, flexible programmatic interfaces combined with strong foundation models should be the backbone of enterprise automation.

1 Introduction

The introduction argues that direct terminal and filesystem agents can automate enterprise workflows through flexible APIs without requiring heavy GUI or tool abstractions. Across diverse production-grade platforms, the paper evaluates this claim and finds that minimal terminal agents match or outperform more complex architectures at lower cost.

Motivation

  • Enterprise automation requires agents to perceive system state, reason about business context, and modify operational data reliably over long horizons.Errors can cascade across records, approvals, and downstream workflows in production systems.Source: §1 · p. 1
  • GUI agents rely on brittle action chains, whereas curated tool registries restrict expressivity; direct APIs instead preserve flexible programmatic interaction.Enterprise platforms already expose APIs for querying data, updating records, and performing complex operations, so additional wrappers may reduce flexibility.Source: §1 · p. 1

Empirical evaluation

  • Minimal terminal agents match or outperform GUI-driven and tool-augmented agents across diverse enterprise workflows while maintaining competitive efficiency.The study compares all three paradigms on realistic production-grade platforms and positions the terminal agent as the foundation for enterprise automation.Source: §1 · p. 2

Benchmark

  • The paper introduces a unified benchmark spanning multiple production platforms, verified evaluation environments, and datasets for realistic enterprise tasks.Source: §1 · p. 2

Extensions

  • The study examines reusable self-created skills and browser access as practical extensions for terminal agents handling genuinely UI-bound tasks.Source: §1 · p. 2

2 Related Work

Prior work contrasts web/GUI agents, which struggle with long enterprise interaction chains, with API-first and coding-agent approaches that offer more compact, reliable interfaces. Related research also externalizes context through tools, filesystems, and persistent memory, a perspective this work applies using filesystem and API interaction.

Web and GUI agents in enterprise benchmarks

  • Web and GUI benchmarks show that enterprise agents remain below human performance on long-horizon tasks, with planning, reasoning, and retrieval as key bottlenecks.WebArena (Zhou et al., 2024), WorkArena, and its compositional extension (Boisvert et al., 2024) establish these limitations across realistic enterprise workflows, alongside SCUBA (Dai et al., 2025) and TheAgentCompany (Xu et al., 2025a).Source: §2 · p. 3

API-first and coding-agent approaches

  • API-first and coding-agent studies find that programmatic interfaces improve reliability, completion time, and task execution by letting agents compose executable operations directly.Beyond Browsing (Song et al., 2025) reports API-enabled agents outperform browsing-only agents, with hybrid agents performing best; AXIS (Lu et al., 2025), CodeAct (Wang et al., 2024a), AppWorld (Trivedi et al., 2024), and Gorilla (Patil et al., 2023) develop related API- and code-based approaches.Source: §2 · p. 3

Externalizing context through environment interaction

3 StarShell

StarShell evaluates whether minimal terminal agents can automate enterprise tasks through direct platform APIs, comparing them with GUI-driven web and MCP tool-use agents under matched backbones and benchmarks. The section describes StarShell’s dynamic API interaction, enterprise environments, and controlled evaluation design.

3 StarShell

  • The study compares web, MCP tool-use, and terminal agents using the same LLM backbone and identical enterprise benchmarks to isolate interaction modality.The evaluation spans multiple frontier LLMs, with each comparison sharing one backbone for fairness.Source: §3 · p. 3, §3.4 · p. 5

3.1 Agent Interaction Paradigms

  • StarShell operates through a terminal and filesystem, writing and executing code to interact directly with platform APIs rather than invoking predefined tools.The filesystem stores documentation, intermediate results, and reusable skills, while execution outputs support iterative reasoning and correction.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 StarShell: A Terminal-Based Enterprise Agent

  • Unlike tool-based agents, StarShell discovers capabilities from documentation and API responses, enabling operations absent from curated tool registries.This flexibility supports composing platform actions that predefined schemas may not represent.Source: §3.2 · p. 4

3.3 Enterprise Benchmark Environments

  • The benchmarks cover ServiceNow, GitLab, and ERPNext, with tasks ranging from record queries to multi-step workflows involving filtering, conditional updates, and multiple objects.ServiceNow and GitLab extend prior tasks with live programmatic verification, while ERPNext introduces a new multi-record enterprise-workflow benchmark.Source: §3.3 · pp. 4–5

3.4 Experimental Setup

  • Experiments first compare interaction paradigms without documentation or reusable skills, then measure documentation access and skill persistence through controlled ablations.Success rate is the primary outcome metric, while inference cost is the primary efficiency metric because it is less sensitive to infrastructure latency.Source: §3.4 · pp. 5–6

4 Experiments

Across three enterprise platforms and four base models, terminal agents provide the strongest cost–performance tradeoff, matching or exceeding web-agent accuracy while substantially outperforming curated tool-use agents. Persistent skills improve terminal-agent performance, while browser fallback helps only selectively despite substantial task complementarity.

4.1 Comparing types of agents

  • Terminal agents match or exceed web-agent accuracy in 8 of 12 platform–model combinations while consistently costing less, and they outperform tool-use agents on every platform.With Gemini 3.1 Pro, terminal agents average 77.5% success across environments at $0.09 per task; web agents can be 9× costlier for only modest accuracy gains, while tool-use agents are constrained by rigid endpoint coverage.Source: §4.1 · pp. 6–7

4.2 Access to self-generated Skills

  • Persistent skills improve success across all platforms, with the largest gain on ERPNext (+5.3pp), while reducing average cost on ServiceNow by 43.7% and ERPNext by 23.9%.Benefits vary with platform familiarity: memory helps most on less familiar ERPNext, modestly on ServiceNow, and only marginally on GitLab, where costs rise slightly and the skills directory remains minimal; magnitudes depend on task order.Source: §4.2 · p. 8

4.3 API-First Agents with Browser Fallback

  • On ServiceNow, an oracle could reach 89.1% by selecting per-task between terminal and web agents, but hybrid execution reaches 83.0% only with Opus 4.6 and performs worse than either alone with Sonnet 4.6.The Opus hybrid costs $2.57, below web’s $4.21 but above terminal’s $1.94; the Sonnet hybrid reaches 72.1% at $3.02, versus 73.6% for terminal and 72.4% for web.Source: §4.3 · pp. 8–9, Table 3 · p. 8

5 Analysis and Discussion

The analysis finds that terminal-agent failures stem mainly from difficulty making task progress, while browser access is advantageous for a minority of UI-bound or indirectly exposed tasks. It attributes the terminal advantage to general interfaces, runtime skills, and terminal-first tool selection, while showing how agents accumulate platform-specific procedural knowledge.

5.1.1 Failure is not driven by tool unreliability

  • Successful and failed tasks encounter comparable API, curl, and shell-error rates, indicating that failures reflect inability to make progress rather than unreliable tools.Tool-call distributions are broadly similar, although some failed tasks stop near zero or enter unproductive loops above 30 calls.Source: §5.1.1 · p. 9, Figure 5 · p. 9, Figure 4 · p. 9

5.1.2 Where the browser is the shorter path

  • Browser access is a shorter path for a minority of tasks, including cookie-dependent impersonation, rendered chart values, and complex workflow editors lacking reliable public API paths.For dashboard values, terminal agents reconstruct aggregations while web agents directly read rendered state; browser access therefore complements rather than replaces terminal capabilities.Source: §5.1.2 · pp. 9–10

5.2 Where does the terminal advantage come from?

  • A generic api_call tool recovers nearly the entire tool-use gap, while runtime-written skills reduce ServiceNow cost by 43.7% and the terminal advantage spans 34.7 to 40.4 points across four backbones.Tool coverage is the largest factor, tool granularity accounts for most of the remainder, and shell/filesystem support adds ephemeral working memory and multi-step scripting.Source: §5.2 · p. 10

5.3 What skills do terminal agents create?

  • Agents created 38 verified skill files across the three platforms, recording API procedures alongside field mappings, valid values, endpoints, and trial-and-error workarounds.ServiceNow produced 11 larger encyclopedic skills, ERPNext 25 smaller entity-focused skills, and GitLab 2 files capturing non-obvious API conventions.Source: §5.3 · pp. 10–11

5.4 How do hybrid agents choose their tools?

  • Across 330 ServiceNow tasks, the hybrid agent made 82% of tool calls through the browser, using the terminal chiefly for record creation and the browser for other categories.This choice succeeds for impersonation and chart reading but can be inefficient: filtering may require 30 to 50 browser calls versus a few API calls with sysparm_query.Source: §5.4 · p. 11
  • The practical strategy is terminal-first with browser fallback, provided the model can route subtasks effectively; platform providers should expose API parity for actions available through the interface.This preserves browser benefits for directly UI-reached tasks without incurring its higher cost and longer trajectories on tasks the terminal handles efficiently.Source: §5.4 · p. 11

6 Conclusion

Across realistic enterprise benchmarks, minimal terminal-and-filesystem coding agents using platform APIs matched or outperformed more complex architectures while remaining efficient. The conclusion recommends API-first agents as the default baseline and identifies long-horizon, cross-platform, human-supervised automation as a key direction for future evaluation.

Conclusion

  • Minimal coding agents using a terminal, filesystem, and direct platform APIs matched or outperformed GUI-driven web agents and MCP-based tool-augmented agents across realistic production-grade benchmarks while remaining competitive on efficiency.The comparison supports using a simple terminal-and-filesystem agent as the default baseline when a platform exposes a sufficiently expressive API.Source: §6 · p. 11
  • Stable programmable interfaces may benefit enterprise automation more than additional abstraction layers because code and API calls remain close to the modalities on which language models are most heavily trained.Browser access remains useful when the UI provides a more direct route, but as an extension to the API-first baseline rather than the primary interface.Source: §6 · p. 11

Future work

  • Future benchmarks should evaluate long-horizon agents that coordinate across platforms, maintain state over extended interactions, incorporate human oversight, and cover IT operations, HR, security, and finance.These properties and verticals are needed to understand how agent architectures scale.Source: §6 · p. 11

Limitations

The central claim is limited to enterprise platforms with expressive APIs, while several important capabilities and deployment conditions remain outside the evaluation. The study also leaves open questions about tool implementations, failure severity, and hybrid routing policies.

Scope of the central claim

  • Terminal agents are the recommended default only for platforms with expressive APIs, while dashboard-chart tasks, impersonation flows, and complex UI-authored artifacts remain more cumbersome or uncovered.In ServiceNow, 40 of 330 tasks involve dashboard-chart values and 10 require session-based impersonation; Flow Designer workflows do not appear in the task set, although terminal agents still lead across all four backbones with those tasks included.Source: Limitations · p. 12

Benchmark scope

  • The benchmark covers three production-grade platforms and 729 single-session tasks but excludes long-horizon workflows, cross-platform coordination, and scenarios requiring human oversight.Building comparable benchmarks for these settings requires fully functioning applications supporting GUI, API, and tool-based evaluation on identical tasks.Source: Limitations · p. 12

Tool-use comparison scope

  • Tool-use findings characterize the evaluated MCP servers, whose coverage ranges from 7 to 83 tools, rather than the full space of implementations.A field-complete server could narrow the observed gap, while the reported analyses use feasible subsets and a minimal single-tool variant to isolate tool granularity.Source: Limitations · p. 12

Outcome quality beyond success rate

  • Binary success rates do not reveal whether terminal, tool-use, and web-agent failures differ in severity or produce incorrect writes and partial side effects.Failure risk, not only failure frequency, remains important for safe deployment.Source: Limitations · p. 12

Hybrid agent routing

  • The hybrid agent relies on the base model’s implicit routing judgment, overuses the browser relative to its oracle upper bound, and does not test learned or rule-based policies.This leaves the effectiveness of alternative routing strategies unevaluated.Source: Limitations · p. 12

A Reproducibility Statement … C.1 Fairness of paradigm comparison

The paper supports reproducibility through released evaluation artifacts, deterministic scoring, controlled environments, and recorded traces, while cautioning that terminal-agent capability results do not establish production readiness. On feasible task subsets, terminal and web agents still outperform tool-use agents, largely because narrow tool interfaces restrict operations and parameters.

A Reproducibility Statement

  • The evaluation framework, datasets, environments, prompts, documentation, and deterministic scoring scripts will be released, with task definitions and success criteria fully specified.Prompts are included in Appendix K and will be released with the source code.Source: A Reproducibility Statement · p. 17
  • Repeated runs show success-rate standard deviations of 0.5–1.4 percentage points on ServiceNow and 1.9–2.2 on GitLab, supporting stable principal comparisons while warranting caution about small differences.The repeated-run analysis covers only a subset of the full evaluation matrix.Source: A Reproducibility Statement · p. 17

B Ethics and Safety Considerations

  • Terminal-agent safety depends more on credential and environment controls than shell allowlisting, and the results address capability and cost rather than readiness for unsupervised production use.The study does not run adversarial tasks or characterize whether failures differ in severity across paradigms; proposed controls include scoped credentials, containers, restricted egress, proxies, auditing, and isolated persisted skills.Source: B Ethics and Safety Considerations · pp. 17–18

C.1 Fairness of paradigm comparison

  • Tool-use agents face structural infeasibility because MCP servers omit operations such as ServiceNow catalog ordering, ERPNext deletion, and several GitLab social and milestone tasks.These omissions affect 210 of 330 ServiceNow tasks and all ten ERPNext deletion tasks.Source: §C.1 · p. 18
  • Terminal agents achieve the highest overall success rate in 3 of 4 model configurations and remain most cost-efficient in 3 of 4 feasible-task comparisons.They tie web agents overall with Claude Opus 4.6, while restricting evaluation to task types feasible for all three paradigms.Source: §C.1 · p. 18, Table 4 · p. 19
  • Sonnet tool-use success on ServiceNow rises from 11.5% to 24.2% on the feasible subset, narrowing—but not eliminating—the gap across models.The comparison retains 12 of 33 ServiceNow task types, 15 of 28 GitLab types, and 78 of 82 ERPNext types.Source: §C.1 · p. 18
  • Even available tools can restrict fields and parameters, whereas terminal and web agents can construct arbitrary payloads or interact with any form element.A create_incident tool may expose only a few parameters although the underlying API accepts dozens of fields.Source: §C.1 · p. 18
  • Replacing a catalog of narrow tools with one generic tool recovers most of the gap, indicating that tool granularity—not MCP itself—drives the limitation.Fixed catalogs expose only explicitly listed operations and parameters, while terminal agents can explore endpoints, adapt payloads, and script complex sequences.Source: §C.1 · pp. 18–19

C.2 Disentangling Tool Granularity from Terminal Access … D Task Generation

Across controlled ablations and model-scale comparisons, flexible direct API access explains most of terminal agents’ advantage, while filesystem capabilities, documentation quality, and orchestration affect performance by task and platform. Terminal agents generalize to smaller open-weight models and remain competitive with more complex systems at lower cost.

C.2 Disentangling Tool Granularity from Terminal Access

  • 73.5% success at $0.15 with Sonnet and 79.2% at $0.25 with Opus show that a single generic API-call tool can match or exceed terminal agents at lower cost.The API-call agent has arbitrary authenticated REST access but no filesystem or shell; its results indicate tool granularity explains most of the gap over curated tool-use agents, while terminal capabilities remain useful for storage, batching, and broader protocols.Source: Table 5 · p. 20, §C.2 · p. 20

C.3 Generalization to open-weight models

  • Terminal agents achieve the highest success rate in every open-weight platform–model combination: 46.1% with Gemma and 54.5% with Qwen, versus lower web and tool-use rates.They are approximately 4.9× cheaper than web agents with Gemma and 3.6× cheaper with Qwen, although tool-use remains least expensive in every comparison.Source: Table 6 · p. 21, §C.3 · p. 21

C.4 Parametric knowledge vs. documentation

  • Documentation provides no clear overall benefit: without documentation, Sonnet reaches 72.7% versus 71.8% with it, while Opus reaches 78.7% versus 79.2%.Effects diverge by platform: documentation hurts ServiceNow, helps ERPNext, and raises GitLab cost without improving accuracy; platform familiarity and documentation structure remain competing explanations.Source: Table 7 · p. 22, §C.4 · p. 22

C.5 How do terminal agents leverage documentation?

  • Documentation helps when task-oriented but can mislead when reference-oriented: ERPNext exposes actionable entity and field guidance, whereas ServiceNow encourages complex retrieval-heavy strategies.This motivates consumer-structured documentation and Skills as complements to static documentation.Source: §C.5 · pp. 22–23

C.6 Single agent vs. multi-agent systems

  • The planner-executor reaches 74.8% versus 72.7% with Sonnet but costs 43% more, while Opus produces nearly identical accuracy at higher multi-agent cost.With Opus, the single agent remains cheaper at $1.22 versus $1.41, so single-agent execution is consistently more cost-efficient overall.Source: Table 8 · p. 23, §C.6 · p. 23

C.6.1 When does multi-agent orchestration help?

  • Planner-executor gains are task- and platform-dependent: Sonnet improves ERPNext by +6.8pp, but the single agent wins on ServiceNow and Opus regresses on GitLab.Planning overhead can be counterproductive when direct execution suffices, suggesting orchestration should be invoked conditionally.Source: §C.6.1 · p. 23
  • On ERPNext, 13 of the planner-executor’s 24 improvements occur on hard tasks requiring linked records and dependent subtasks, where schema exploration resolves field ambiguities before execution.The benefit nearly disappears with Opus, suggesting explicit orchestration primarily compensates for weaker models’ difficulty reasoning over complex task structures.Source: §C.6.1 · pp. 23–24

D.1 Task Design … F Full Results

The evaluation spans redesigned ServiceNow and GitLab tasks plus a newly constructed ERPNext benchmark, using state-based validation and a taxonomy of curl tool-call outcomes. Results compare agent paradigms only within this evaluation pipeline.

D.1 Task Design

  • Hard ERPNext templates chain dependent subtasks, while remaining templates target single operations, mirroring WorkArena++ compositional levels L2 and L3 (Boisvert et al., 2024).Chained goals can require multiple linked records whose identifiers or field values depend on earlier creations.Source: §D.1 · p. 24
  • Tasks cover 330 ServiceNow, 192 GitLab, and 207 ERPNext instances across operations including creation, retrieval, updates, deletion, filtering, sorting, navigation, and composite workflows.ERPNext is newly built with linguist-authored seed data, goals, and SQL validators; ServiceNow and GitLab adapt prior benchmarks while redesigning evaluation.Source: §D.1 · p. 24
  • Every task requires navigation from its natural-language goal without a start URL, and descriptions remove ambiguity because agents lack browser context.These changes make the tasks harder to locate and more precisely specified than the original benchmarks.Source: §D.1 · p. 24
  • Comparisons are limited to agent paradigms within the rebuilt evaluation pipeline because ServiceNow and GitLab tasks and scoring are not directly comparable to published WorkArena or WebArena results.Tasks are scored against platform state rather than the original benchmarks’ criteria.Source: §D.1 · p. 24

D.2 Evaluation

  • Validators reset each platform to a fixed snapshot, check pre- and post-execution state, and verify expected outcomes through live-instance API calls or ERPNext SQL queries.This prevents task interference and ensures tasks cannot pass without agent action.Source: §D.2 · p. 24
  • For read-oriented tasks, validators extract the final answer or URL from the agent’s message and compare it with ground truth rather than relying on string matching alone.This evaluates retrieval and navigation tasks against actual platform state despite their lack of write side effects.Source: §D.2 · p. 24

E Taxonomy of tool call outcomes

  • Table 9 classifies curl tool-call outcomes from web-platform evaluation traces into success or failure types based on response content.The supplied table passage identifies the taxonomy’s scope but does not provide category counts or comparative results.Source: Table 9 · p. 25

F.1 Full Results: Comparing Agent Interaction Paradigms … F.5 Full results: Hybrid Agents

The full-results experiments compare terminal, web, tool-use, skill-enabled, multi-agent, and hybrid interaction paradigms using success, efficiency, time, and cost metrics. Across the reported subsections, evaluations span enterprise platforms and backbone models while varying documentation access, skill reuse, agent composition, and tool integration.

F.1 Full Results: Comparing Agent Interaction Paradigms

  • Table 10 reports success rate, tool calls, wall-clock time, and cost for tool-use, web, and terminal agents across three enterprise platforms and four backbone LLMs.The table identifies the best metric values within each model group and success rates within one standard error of the best.Source: Table 10 · p. 26
  • The paradigm comparison evaluates terminal agents alongside web and tool-use agents using success rate, average tool calls, time, and cost per task.These metrics provide the reported basis for comparing interaction paradigms.Source: Table 10 · p. 26
  • Documentation access is tested by comparing terminal agents with and without official platform documentation across success rate, tool calls, time, and cost.Table 11 reports the best metric values within model groups and success rates within one standard error of the best.Source: Table 11 · p. 26

F.3 Full Results: Agents Equipped with Skills

  • Skill-enabled terminal agents are compared with terminal agents lacking self-generated skill creation and reuse using success, tool-use, time, and cost measures.Table 12 reports these metrics within model groups and marks best values, including success rates within one standard error of the best.Source: Table 12 · p. 27

F.4 Full Results: Single Agent vs Multi-Agent System

  • A single terminal agent is evaluated against a planner-executor multi-agent system whose agents share the same model and tool access.The comparison reports success rate, average tool calls, wall-clock time, and cost per task.Source: Table 13 · p. 27

F.5 Full results: Hybrid Agents

  • The hybrid-agent experiment compares a Playwright MCP-plus-terminal agent with a simple terminal agent and an agent using only Playwright MCP tools.Success rate, tool calls, wall-clock time, and cost per task are reported for each design.Source: Table 14 · p. 27

G Task Types per Benchmark … I Playwright MCP Server

This section catalogs benchmark task types and the tools exposed by the ServiceNow, ERPNext, GitLab, and Playwright MCP servers. It also records implementation and configuration limits that determine which advertised tools were actually available to agents.

G.1 ServiceNow

G.2 GitLab

G.3 ERPNext

H.1 ServiceNow

  • The ServiceNow MCP server exposes 83 tools that the model can actually invoke, although its default package lists 93 and 11 lack implementations.Source: Table 18 · p. 31

H.2 ERPNext

  • The ERPNext MCP server’s available tools are summarized in Table 19, including document-management and schema operations.Source: Table 19 · p. 31, §H.2 · p. 32

H.3 GitLab

  • The GitLab MCP server advertises 81 tools, while 26 additional wiki, milestone, and pipeline tools remain unavailable because their enabling environment variables are unset.Source: Table 20 · p. 33

I Playwright MCP Server

  • The Playwright MCP configuration provides 21 tools, while vision, PDF, tracing, and browser installation capabilities are disabled or blocked and therefore unavailable in recorded trajectories.Source: Table 21 · p. 34, I Playwright MCP Server · p. 34

J Error Analysis Tooling · K Prompts · L Error Analysis

This section documents tooling and prompt designs for analyzing agent traces and guiding MCP, web, terminal, API-call, and hybrid agents across enterprise platforms. It covers persistent skills, planning and execution workflows, tool-use constraints, verification, and user-facing links.

J Error Analysis Tooling

  • The error-analysis interface combines run metrics, per-sample outcomes, reasoning inspection, and detailed agent and environment logs.Figure 6 shows the user interface used to inspect agent traces.Source: J Error Analysis Tooling · p. 35, Figure 6 · p. 35

K.1–K.3 Tool-use Agent Prompts

  • ServiceNow, GitLab, and ERPNext tool-use prompts direct agents to use platform-specific MCP tools, verify record changes, handle errors adaptively, filter listings, and return direct URLs.The prompts expose domain-specific capabilities spanning incident and change management, project and repository workflows, accounting, inventory, HR, manufacturing, and CRM.Source: §K.1 · p. 36, §K.2 · p. 37, §K.3 · p. 38

K.4 Web Agent Prompt for ServiceNow

  • The web-agent prompt equips a Playwright-controlled browser with navigation, interaction, extraction, screenshots, JavaScript execution, and accessibility-based verification for ServiceNow tasks.Authentication headers are preconfigured, and the prompt advises checking page state and handling iframes when needed.Source: §K.4 · p. 39

K.5 Terminal Agent Prompt for ServiceNow

  • The terminal-agent prompt uses bash and authenticated curl requests to query, filter, sort, and modify ServiceNow records while keeping commands short and returning verification URLs.Examples use environment-provided authentication headers, sysparm_query, sysparm_fields, and POST requests.Source: §K.5 · p. 40

K.6 API-call Agent Prompt for ServiceNow

  • The API-call prompt gives an agent one authenticated api_call tool and requires it to construct REST methods, paths, and JSON bodies from its own API knowledge without documentation or a filesystem.It also requires read-back confirmation, adaptive error handling, filtered listings, and reproducible query URLs.Source: §K.6 · p. 41

K.7 Prompt extension for Docs

  • The documentation extension instructs terminal agents to consult a hierarchical docs/ directory before API calls and use shell commands to locate, search, and read endpoint guidance.The documentation files contain YAML frontmatter followed by Markdown content.Source: §K.7 · p. 42

K.8 Prompt extension for Skills

  • The skills extension treats skills/ as persistent memory that agents search before tasks and update afterward with reusable procedures, verified status, examples, failures, and pitfalls.It recommends generalizing procedures, avoiding hardcoded URLs, correcting wrong skills, and maintaining one file per topic in a living knowledge base.Source: §K.8 · pp. 43–44
  • The skills prompt advises updating or creating a skill only when needed and avoiding duplication of existing documentation.Source: §K.8 · p. 44

K.9 Prompts for the Planner and Executor in the Multi-Agent System

  • The planner researches the live ServiceNow instance through read-only GET requests, avoids state changes, and produces a detailed numbered plan with endpoint and field details for the executor.The executor follows the plan, adapts when necessary, and provides a direct URL to the completed record or page.Source: §K.9 · p. 45

K.10 Prompt for the Hybrid Agent

  • The hybrid prompt assigns terminal tools to API and data operations, browser tools to UI-specific tasks, and requires switching tools when appropriate plus live verification and reproducible result links.It specifies ServiceNow API examples, prefers terminal operations for record management, and includes browser practices such as accessibility inspection, state checks, screenshots, and iframe handling.Source: §K.10 · pp. 46–47

L.1 Case Study: Documentation Leading to a Suboptimal API Strategy … L.4 Case Study: Planner-Executor Resolving Ambiguous Field Semantics

The case studies show that documentation, skills, and planner-executor designs affect terminal-agent success in task-dependent ways: documentation can mislead API strategy or enable conceptual field discovery, while skills reduce exploration overhead and planning resolves ambiguous semantics.

L.1 Case Study: Documentation Leading to a Suboptimal API Strategy

  • Documentation caused failure by steering the agent to a two-step cart workflow that omitted required variables, whereas direct exploration found the atomic order_now endpoint and succeeded.The documented workflow produced a valid request number but failed validation because catalog variables were not persisted; without documentation, the agent succeeded with score = 1 in 7 tool calls.Source: §L.1 · p. 48
  • The documentation-induced ServiceNow failure arose because add_to_cart handled only the base item, requiring an additional undocumented variable-setting call that the agent did not perform.The documentation was technically correct for general-purpose cart operations but practically misleading for this task because it omitted the simpler atomic endpoint.Source: §L.1 · p. 48

L.2 Case Study: Documentation Enabling Correct Field Discovery

  • Documentation enabled correct ERPNext configuration by revealing the conceptual meaning of an option, prompting schema lookup and setting so_required = 1; the task completed successfully in 27 tool calls.Without documentation, the agent left the counterintuitive field at so_required = 0 and failed evaluation; documentation supplied conceptual scaffolding rather than the exact API call or field name.Source: §L.2 · p. 49

L.3 Case Study: Skills Reducing Exploration Overhead

  • Encoding inherited field mappings and a file-based payload pattern reduced ServiceNow cost to $0.34 from a baseline of 8 tool calls at $0.51, a 33% reduction.The three saved calls avoided two failed discovery calls and one failed inline attempt through inherited impact-field mappings and a file-based payload pattern.Source: §L.3 · p. 50
  • Skills amortized API exploration across tasks by encoding that impact choices are inherited from the task table and that multiline JSON should be written to a file before posting.The baseline encountered missing direct impact choices and a shell-escaping error, while the skilled agent looked up only the instance-specific assignment-group identifier and succeeded on its first payload attempt.Source: §L.3 · p. 50

L.4 Case Study: Planner-Executor Resolving Ambiguous Field Semantics

  • The planner read the field’s human-readable label, mapped the requirement to so_required = 1, and enabled correct execution, while the single agent scored 0 after misinterpreting inverted semantics.The executor followed the plan in 8 tool calls and set the field correctly without independently reasoning about its semantics.Source: §L.4 · p. 51
  • Planner-executor is especially useful when programmatic field names obscure semantic intent, because schema exploration can expose human-readable labels before execution.In the ERPNext example, the planner’s structured output acted as documentation for the executor by explicitly mapping the task requirement to so_required = 1.Source: §L.4 · p. 51

Related papers

Loading 2604.00073v3…