PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal Agents Suffice for Enterprise Automation

Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar

arXiv:2604.00073v3cs.SEcs.AIcs.CL

TL;DR

Enterprise automation requires agents to act across production systems, but it remains unclear whether GUI and curated-tool abstractions are necessary when stable APIs are available. The paper compares these paradigms with minimal terminal agents across realistic enterprise benchmarks and finds that terminal agents match or outperform more complex architectures while remaining more efficient. It concludes that terminal and filesystem access should be the default foundation where expressive APIs exist, with extensions for tasks that need them.

  • Problem

    It remains unclear whether additional GUI and curated-tool abstraction layers are necessary for enterprise automation when stable APIs are available.

  • Method

    The paper compares GUI-driven, MCP tool-augmented, and minimal terminal coding agents across realistic tasks on multiple production platforms using systematic benchmarks.

  • Results

    Terminal agents match or outperform web agents and MCP tool-use agents across diverse workflows, matching or exceeding web accuracy in 8 of 12 platform–model combinations while consistently costing less.

  • Takeaways & Limitations

    A terminal and filesystem should be the default foundation for enterprise automation on platforms with expressive APIs, extended with skills or browser access when required.

  • Takeaways & Limitations

    The evaluation covers three platforms and 729 single-session tasks but not long-horizon multi-session workflows, cross-platform coordination, or human-oversight scenarios.

Abstract

from arXiv · show

There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously. Among the approaches explored are tool-augmented agents built on abstractions such as Model Context Protocol (MCP) and web agents that operate through graphical interfaces. Yet, it remains unclear whether such complex agentic systems are necessary given their cost and operational overhead. We argue that a coding agent equipped only with a terminal and a filesystem can solve many enterprise tasks more effectively by interacting directly with platform APIs. We evaluate this hypothesis across diverse real-world systems and show that these low-level terminal agents match or outperform more complex agent architectures at a fraction of the cost. Our findings suggest that simple, flexible programmatic interfaces combined with strong foundation models should be the backbone of enterprise automation.

1 Introduction

Enterprise agents increasingly need to execute long-horizon actions on production systems, but GUI and curated-tool abstractions impose brittleness or expressivity limits. This work tests whether minimal terminal agents using direct APIs can provide a simpler, effective alternative.

  • Motivation: Enterprise agents must perceive system state, reason about business context, and modify operational data across long-horizon workflows.Errors can cascade across records, approvals, and downstream workflows.
  • Motivation: GUI agents use brittle action chains, while curated tool registries restrict execution to predefined operations and compositions.These tradeoffs motivate testing more flexible programmatic interaction.
  • Research question: The study compares GUI-driven, tool-augmented, and minimal terminal-based agents across realistic workflows on multiple production-grade platforms.The terminal agents interact through direct APIs and are evaluated against more complex architectures.
  • Findings: Terminal agents match or outperform more complex architectures while maintaining competitive efficiency across diverse enterprise workflows.The paper therefore positions the coding agent as a foundation that can be extended with persistent skills or browser access when needed.
  • Contributions: The paper introduces a unified benchmark spanning multiple production platforms, verified evaluation environments, and realistic enterprise-task datasets.It also studies reusable skills and browser access as extensions for terminal agents.

2 Related Work

Prior work studies GUI-based enterprise agents, API-first interaction, and environment-mediated context externalization. These lines of research motivate direct programmatic interfaces as an alternative or complement to browser control and curated tools.

  • Web and GUI agents: Web-agent benchmarks show that strong agents remain well below human end-to-end success, especially on tasks requiring long interaction chains.WebArena and WorkArena provide realistic environments for studying such tasks.
  • API-first approaches: API-enabled agents outperform browsing-only agents, while hybrid API-and-browsing agents perform best in related evaluations.This supports programmatic interaction as a compact alternative to step-by-step UI navigation.
  • Externalized context: REPL-style and software-oriented agents externalize context through environment interaction rather than relying only on a single prompt.This line includes Recursive Language Models, SWE-agent, and OpenHands.

3 StarShell

The study evaluates three interaction modalities on realistic enterprise tasks using controlled environments and shared model backbones. StarShell is the minimal terminal agent: it writes and executes code, uses the filesystem, and discovers APIs dynamically rather than relying on predefined schemas.

  • Experimental design: The experiment compares GUI-driven web agents, MCP tool-augmented agents, and minimal terminal agents while holding the LLM backbone constant.The evaluation isolates interaction modality before testing documentation access and reusable skills.
  • Agent paradigms: Tool-augmented agents invoke curated MCP operations, simplifying execution but constraining agents to the functionality exposed by the registry.Examples include listing records, creating entries, and updating fields.
  • Agent paradigms: Web agents observe rendered graphical interfaces and issue low-level actions such as clicking, typing, and navigating pages.The implementation uses Playwright MCP tools for page inspection, interaction, and state verification.
  • StarShell: StarShell operates through a terminal and filesystem, writing and executing code to query APIs, manipulate data, and inspect execution outputs iteratively.Its primary interfaces are command execution and filesystem artifact storage.
  • StarShell: StarShell discovers platform capabilities from documentation or API responses and composes operations absent from curated tool registries.This dynamic discovery enables flexible API interaction and data manipulation.
  • Benchmark environments: The benchmark covers ServiceNow, GitLab, and ERPNext, with natural-language tasks spanning queries, record modifications, filtering, conditional updates, and multi-object workflows.The environments provide sandboxed data, API access, optional MCP registries, and local documentation.
  • Metrics: Success rate measures verified task completion, while inference cost measures token usage as the primary efficiency metric.Cost is preferred over wall-clock time because it is less sensitive to infrastructure latency.

4 Experiments

Across enterprise benchmarks, terminal agents provide the strongest cost–performance tradeoff: they match or exceed web-agent accuracy while costing less, and outperform curated tool-use agents. Persistent skills improve success and often reduce cost, while hybrid browser access does not consistently justify its added expense.

  • Agent comparison: Tool-use agents achieve the lowest success rates across models, especially on ServiceNow at 11.5–18.5%, because curated endpoints may not support required actions.ERPNext’s seven general-purpose CRUD tools outperform the much larger narrow catalogs on ServiceNow and GitLab, showing that catalog size alone does not determine performance.
  • Agent comparison: Web agents achieve the highest or tied-highest accuracy in 8 of 12 combinations but process costly accessibility trees and screenshots.On ERPNext with Opus 4.6, web costs $6.49 per task versus $0.72 for terminal, a 9× difference for a modest accuracy gain.
  • Agent comparison: Terminal agents match or exceed web-agent accuracy in 8 of 12 platform–model combinations while consistently costing less.On the full benchmark, the two paradigms are within about a point for three of four models, making the main advantage cost rather than accuracy.
  • Self-generated skills: Persistent memory improves success rate across all three platforms, with gains of +5.3pp on ERPNext, +3.4pp on ServiceNow, and +1.6pp on GitLab.The largest gain occurs where tasks involve unfamiliar field names and multi-step dependencies; straightforward API calls benefit less.
  • Self-generated skills: Memory reduces average cost by 43.7% on ServiceNow ($0.44 vs. $0.78) and 23.9% on ERPNext ($0.35 vs. $0.46), but raises GitLab cost ($0.31 vs. $0.28).The skills directory is checked in 1–2 calls, and new entries are written only for genuinely novel information.
  • Browser fallback: Hybrid access has an 89.1% oracle ceiling on ServiceNow, but Sonnet 4.6 reaches only 72.1%, below Terminal at 73.6% and Web at 72.4%, while costing $3.02.With Opus 4.6, hybrid reaches 83.0%, the highest configuration, at $2.57—less than Web’s $4.21 but more than Terminal’s $1.94.

5 Analysis and Discussion

The analysis attributes terminal agents’ advantage to flexible interfaces, while browser access remains valuable for a minority of UI-bound or browser-state tasks. Failures more often reflect stalled progress than unreliable tools, and hybrid systems work best with disciplined routing.

  • Failure is not driven by tool unreliability: Comparable API, curl, and shell error rates across successful and failed tasks indicate that failures are driven by stalled task progress rather than tool unreliability.Failed tasks cluster near zero tool calls or show occasional unproductive loops above 30 calls.
  • Where the browser is the shorter path: Browser access is the shorter path for browser-session operations such as ServiceNow impersonation, which depends on cookies that the terminal agents did not reproduce.The web agent completes impersonation through two menu interactions after the terminal agent receives HTTP 200 without changing session state.
  • Where the browser is the shorter path: UI-rendered values are programmatically reachable but indirect because recomputation can differ from displayed values through rounding, formatting, or chart-specific logic.The web agent directly observes the rendered chart state.
  • Hybrid access: The study supports a terminal-first, browser-as-fallback design because browser access is most valuable for a minority of tasks with a more direct UI route.Routing quality depends on the model selecting the appropriate interface, and skills may help encode that guidance.
  • Where does the terminal advantage come from?: Tool coverage is the largest contributor to the terminal advantage, while a generic api_call tool recovers nearly the entire remaining gap.Shell and filesystem access add ephemeral working memory and multi-step scripting; error recovery and token overhead do not explain the gap.
  • Reusable skills: Across platforms, agents created 38 verified skill files that preserve platform-specific field mappings, valid values, endpoints, and workarounds discovered through trial and error.ServiceNow skills average 3.9 KB, while ERPNext has 25 files averaging 1.8 KB; GitLab has two files capturing undocumented conventions.
  • Reusable skills: Skills also capture infrastructure-level shell-quoting lessons, including avoiding inline JSON with eval curl and using temporary files or scripts instead.This lesson generalizes beyond individual platform tasks.
  • Hybrid access: Hybrid agents use the browser for most calls and can handle impersonation and dashboard reading, but browser-heavy filtering and sorting become substantially more expensive than terminal queries.On ServiceNow, 82% of hybrid tool calls are browser-based; filtering can require 30 to 50 browser calls versus a few API calls.

6 Conclusion

Across realistic benchmarks on production-grade platforms, minimal terminal-and-filesystem coding agents match or outperform GUI and MCP-based architectures while remaining competitive on efficiency. The conclusion recommends them as the default foundation when APIs are sufficiently expressive, with browser access as an extension for more direct UI routes.

  • Conclusion: Minimal coding agents using a terminal, filesystem, and direct APIs match or outperform GUI-driven and MCP-based agents across realistic enterprise benchmarks.The paper frames this simplicity as keeping code and API calls close to the modality on which LLMs are heavily trained.
  • Conclusion: Terminal-based API-first agents should be the default baseline when platforms expose sufficiently expressive APIs, while browser access remains an extension for more direct UI routes.Future evaluation should cover long-horizon, cross-platform, and human-oversight settings across additional enterprise verticals.

Limitations

The central claim is scoped to platforms with expressive APIs and does not imply equal convenience for every task. Important boundaries include excluded long-horizon and cross-platform workflows, limited MCP coverage, unmeasured failure severity, and unevaluated routing policies.

  • Scope of the central claim: Terminal agents are presented as the default foundation for platforms with expressive APIs, not as equally convenient for every task.Dashboard-chart values and complex UI-authored artifacts are identified as cases where browser access can be more direct.
  • Scope of the central claim: Complex UI artifacts such as Flow Designer workflows represent a capability gap, but they are uncommon in practice and absent from the benchmark task set.The paper therefore does not establish terminal-agent performance on this class of task.
  • Benchmark scope: The evaluation covers three production-grade platforms and 729 single-session tasks but excludes long-horizon multi-session workflows, within-task cross-platform coordination, and human oversight.Such benchmarks require functioning applications that expose GUI, API, and tool-based interfaces for the same tasks.
  • Tool-use comparison scope: The MCP comparison characterizes the evaluated off-the-shelf servers, whose tool coverage ranges from 7 to 83 tools, rather than the full space of MCP implementations.A field-complete server could narrow the observed gap.
  • Outcome quality beyond success rate: Success rate is binary, so the study does not characterize whether failures differ in severity, including incorrect writes or partial side effects across paradigms.Failure risk, not only frequency, matters for safe deployment.
  • Hybrid agent routing: The hybrid agent relies on implicit model judgment for routing and overuses the browser relative to the oracle upper bound, without evaluating learned or rule-based routing policies.The reported hybrid behavior therefore does not establish the performance of explicit routing strategies.

A Reproducibility Statement

The evaluation framework releases its materials and uses deterministic scoring, while explicitly documenting uncertainty, safety boundaries, and fairness concerns in the comparisons.

  • Reproducibility: The authors will release the evaluation framework, datasets, environments, prompts, and documentation, with deterministic metrics and scoring scripts given model outputs.
  • Uncertainty: Uncertainty is quantified through sample-proportion standard errors and targeted three-run evaluations that measure stochastic variation across agent paradigms.For success rates near 80%, standard errors are approximately 2.2, 2.9, and 2.8 percentage points on the three platforms, and 1.5 points overall.
  • Safety: Terminal agents can directly modify enterprise records through APIs, so role-scoped credentials, read-only access, and platform auditing are treated as meaningful controls.In ServiceNow error analysis, 97.3% of tool calls were curl requests against the REST API.
  • Safety: Terminal agents expose a more expressive but less constrained action space, while typed tools retain advantages in schema constraints, permission control, and auditability.
  • Scope and fairness: The study evaluates interaction abstractions rather than adversarial safety, failure severity, or every feasible task type equally across paradigms.Some ServiceNow tasks are structurally infeasible for the evaluated tool catalog, so a feasible-subset comparison is also reported.
  • Scope and fairness: On the feasible subset, tool-use performance improves, but terminal and web agents still outperform tool-use agents consistently.Sonnet tool-use on ServiceNow rises from 11.5% to 24.2%, while terminal agents remain the most cost-efficient in 3 of 4 model configurations.

C.2 Disentangling Tool Granularity from Terminal Access

An API-call ablation separates flexible API access from terminal-specific capabilities. It indicates that tool granularity explains much of the tool-use gap, while filesystem and shell capabilities retain narrower practical benefits.

  • Ablation design: The API-call agent uses one authenticated generic HTTP tool that can issue arbitrary methods, paths, queries, and bodies without filesystem or shell access.
  • Results: The API-call agent closes nearly all of the tool-use-to-terminal gap and sometimes matches or exceeds terminal performance.
  • Results: 73.5% success at $0.15 per task gives Sonnet 4.6 API-call agents the highest overall success rate and lowest cost among the compared paradigms.The corresponding terminal values are 72.7% and $0.56, while tool-use reaches 32.9% at $0.51.
  • Interpretation: The ablation identifies tool granularity, rather than terminal access alone, as explaining most of the tool-use-to-terminal gap.Terminal-specific residual advantages include filesystem-backed storage, batched execution, documentation retrieval, persistent Skills, and multi-step scripting.
  • Generalization: Terminal agents retain the highest success rate across open-weight model and platform combinations, while remaining approximately 4.9× cheaper than web agents with Gemma and 3.6× cheaper with Qwen.

D.1 Task Design

The benchmark combines adapted and newly constructed enterprise workflows with compositional difficulty, navigation requirements, platform-state validation, and independent task resets.

  • Task coverage: ERPNext hard templates chain dependent subtasks whose later identifiers or field values depend on earlier records.
  • Task coverage: The benchmark contains 330 ServiceNow tasks across 33 templates, 192 GitLab tasks, and 207 ERPNext tasks covering CRUD, retrieval, filtering, navigation, and composite workflows.
  • Task design: Every task requires navigation because agents receive neither a start URL nor a direct link to the relevant record.
  • Evaluation: The paper reports paradigm comparisons within its own pipeline because its redesigned tasks are not directly comparable to published WorkArena or WebArena results.
  • Evaluation: Each task runs from a fixed restored snapshot, preventing tasks from interfering with one another.
  • Evaluation: Success is evaluated against platform state using pre- and post-trajectory validators, with final-message checks for read-oriented tasks.

F.1 Full Results: Comparing Agent Interaction Paradigms

The results tables report success, tool use, runtime, and cost across three agent paradigms and multiple enterprise platforms, with several ablations examining documentation, skills, planning, and hybrid interaction.

  • Full results: Table 10 compares tool-use, web, and terminal agents across three platforms and four backbone LLMs using success rate, average tool calls, time, and cost.
  • Ablations: Table 11 measures how access to official platform documentation affects terminal-agent success, tool calls, runtime, and cost.
  • Ablations: Table 12 compares terminal agents with and without self-generated reusable skills using the same success, tool, time, and cost metrics.
  • Ablations: Table 13 contrasts a single terminal agent with a planner-executor system in which both agents share the same model and tool access.
  • Ablations: Table 14 compares a terminal-only agent, a Playwright-MCP-only agent, and a hybrid agent combining both interaction modes.

G Task Types per Benchmark

The benchmark inventories enterprise tasks and the tools exposed by the ServiceNow, ERPNext, GitLab, and Playwright environments. It distinguishes task categories from the available tool surfaces across platforms.

  • Tables 15–17 list the ServiceNow, GitLab, and ERPNext benchmark tasks.
  • The task inventory includes document-management and schema operations, while browser coverage includes tab management.
  • 83 tools are available in the ServiceNow MCP server’s actual ListTools response, although its package configuration lists 93.Eleven configured tools lack implementations and are silently omitted.
  • The ERPNext MCP server’s available tools are summarized separately in Table 19.
  • 81 tools are advertised by the GitLab MCP server, while 26 additional tools remain unavailable unless opt-in environment flags are enabled.
  • The Playwright MCP configuration exposes 21 browser tools, with vision, PDF, tracing, and browser-install capabilities disabled or blocked.

J Error Analysis Tooling

The error-analysis interface centralizes outcome metrics, per-sample results, agent reasoning, and execution logs. The compared agents use distinct interaction modes: MCP tools, browser actions, or terminal commands against platform APIs.

  • Figure 6’s interface displays success rate, average tools used, per-sample outcomes, agent reasoning, and agent and environment logs.
  • Agent interfaces: Tool-use agents interact through platform-specific MCP tools organized into categories such as incidents, projects, repositories, accounting, inventory, and workflows.
  • Agent interfaces: Tool-use prompts require selecting the appropriate tool, confirming created or updated records by reading them back, and returning direct platform URLs.
  • Agent interfaces: Web agents use Playwright-controlled browsers to navigate, click, fill forms, select options, inspect page structure, take screenshots, and execute JavaScript.
  • Agent interfaces: Web-agent prompts emphasize accessibility-based interaction, post-action verification, page-state checks, and handling ServiceNow’s iframes.
  • Agent interfaces: Terminal agents use bash and authenticated curl commands to query, filter, sort, and update platform APIs directly.

L.1 Case Study: Documentation Leading to a Suboptimal API Strategy

The case studies show that documentation can either mislead agents toward multi-step API workflows or provide conceptual guidance that prompts schema investigation. Direct API exploration and reusable skills can instead reduce unnecessary exploration and preserve required configuration.

  • ServiceNow case: Without documentation, the agent discovers order_now and succeeds in one call that atomically supplies the item, quantity, and configuration variables.The run receives score = 1 after 7 tool calls.
  • ServiceNow case: With documentation, 12 of 19 tool calls are spent retrieving documentation before the agent follows the two-step ordering workflow.
  • ServiceNow case: A documented add_to_cart→checkout workflow omits ServiceNow’s simpler order_now endpoint, causing the agent to place an incompletely configured order.The documented endpoints create and check out the cart, but variables are not persisted to the resulting request item.
  • ERPNext case: In ERPNext, documentation provides conceptual awareness of a per-customer override, but the agent still queries the Customer schema to identify the field so_required.
  • ERPNext case: Without that conceptual guidance, the agent leaves so_required = 0 unchanged because the counterintuitive field semantics are misunderstood, and the evaluation fails.
  • Skills case: Skills reduce a ServiceNow change-request run from 8 calls at $0.51 to 5 calls at $0.34, a 33% cost reduction.The skill supplies inherited impact mappings and a file-based JSON payload pattern.
  • Planner-executor case: The planner-executor example shows that schema exploration can expose ambiguous field labels and encode their interpretation for subsequent execution.
Loading 2604.00073v3…