Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents

1 Introduction

• The study conducts a controlled comparison of five interfaces on TheAgentCompany and APEX-Agents, covering software-company roles and professional-services work.The interfaces are Tool-only, Bash+Tool, Bash, Bash+Synthesis, and programmatic tool calling (PTC).Source: §1 · p. 1
• Across both benchmarks and models, Bash outscores both shell-free interfaces while remaining among the cheapest options.Adding typed or synthesized tools to Bash yields no detectable pooled score gain, while PTC uses fewer tokens than Tool-only but underperforms Bash in task score and token efficiency.Source: §1 · p. 1
• The paper contributes a controlled comparison of direct tool calling and code-based orchestration on realistic enterprise workflows, evaluating quality and cost across professional roles.Source: §1 · p. 1

2 Related Work

• Code-based approaches express reasoning, actions, or control flow in executable programs, and programmatic tool calling applies this approach to a fixed catalog [Anthropic 2025b; OpenAI 2026b].Source: §2 · p. 1
• SWE-agent, mini-SWE-agent, Live-SWE-agent, and Agentless illustrate specialized tools, shell-only execution, self-authored tools, and fixed workflows, respectively [Yang et al. 2024; Lieret, Yang, and agent Team 2025; Xia et al. 2025; Xia et al. 2024].Tool synthesis and persistent skill libraries also motivate testing whether synthesized tools add value to a shell-based enterprise agent [Cai et al. 2023; Qian et al. 2023; Wang et al. 2023; Yuan et al. 2023; Wang, Fried, and Neubig 2024; Liu et al. 2026; Haque et al. 2025; Wu et al. 2024].Source: §2 · pp. 1–2
• Prior enterprise benchmarks cover customer service, web applications, desktop tasks, and workplace workflows, while Patel et al. (2026) compares programmatic with native JSON tool calling on a BFCL v4 subset.This study compares shell execution, direct tool calling, and programmatic tool calling on realistic enterprise workflows [Xu et al. 2024; Vidgen et al. 2026].Source: §2 · p. 2

3 Method

• A shell permits arbitrary code execution and many actions per step, while a curated catalog limits actions to deployer-authorized functions; PTC chains, loops over, or parallelizes catalog calls.Source: §3 · p. 2
• The five interfaces share the same model, task text, base prompt, and stopping criteria, with interface-specific prompts for synthesis and PTC.Source: §3 · p. 2
• Bash exposes only shell access, Tool-only only typed tools, and Bash+Tool both; Bash+Synthesis adds a persistent directory for reusable tools across sequential tasks.PTC lets the model write Python programs over typed tools in a restricted runtime with no shell or other action channel.Source: §3 · p. 2

4 Experiment Setup

• All five interfaces are evaluated on TheAgentCompany and APEX-Agents using Opus-4.8 and GPT-5.5 at default reasoning levels.Source: §4 · p. 2

4.1 TheAgentCompany

• TheAgentCompany simulates software-company coding and office workflows using shared services, local files, and simulated coworkers.Because it has no LLM-facing typed catalog, the study constructs 60 typed tools for its 174 evaluated tasks; coworker reliability fixes apply uniformly, and checkpoint evaluators grade tasks.Source: §4.1 · pp. 2–3

4 Experiment Setup

• Table 1 defines score as an effort-weighted checkpoint rate for TheAgentCompany or mean rubric score for APEX-Agents, and reports pass rates with 95% bootstrap confidence intervals.Token columns are means per task in thousands, while estimated cost and wall-clock time are also reported per task.Source: Table 1 · p. 3

4.2 APEX-Agents

• APEX-Agents evaluates professional analysis in investment banking, management consulting, and corporate law through shared-workspace document, spreadsheet, and presentation tasks.The evaluation uses all 480 tasks from a fixed June 2026 nine-server snapshot, retains 20 typed application tools, and uses a guardrailed workspace shell; an LLM judge scores rubric criteria from responses and file changes.Source: §4.2 · p. 3

5.1 Core Outcomes

• Bash outscores both shell-free interfaces in every benchmark-model arm while remaining among the cheapest options; PTC scores below Bash throughout.Adding typed or synthesized tools generally increases token use without detectable pooled score improvement.Source: §5.1 · p. 3, Figure 3 · p. 4

5.2 Tool Use

• Bash reduces tool calls and tool-call tokens relative to Tool-only, with tool-call tokens more than halved on TheAgentCompany and similar call-success rates.Tool tokens count tool names, arguments, and returned results, not billed usage or internal Bash/PTC operations.Source: §5.2 · p. 4, Table 2 · p. 4
• PTC has the lowest tool-call success rate on both benchmarks, while Tool-only has the highest exact-repeat rate.The paper suggests that bundling repeated operations may explain the lower repeat rate in shell-based and programmatic interfaces.Source: §5.2 · p. 4

5.3 Paired Performance by Task Complexity

• Bash gains vary with benchmark and task-count proxy: its advantage peaks at intermediate counts on TheAgentCompany, while both interfaces gain more at high counts on APEX-Agents.PTC’s mean score difference becomes positive at high APEX-Agents counts, but Bash retains the larger advantage; Tool-only call count may reflect interface inefficiency as well as task difficulty.Source: §5.3 · p. 4

5.4 Bash Efficiency

• Bash+Tool combines the fewest median shell calls per shell-using task with the highest failure rate and lowest retry rate on both benchmarks.Composition remains common across all three shell interfaces, even when typed or synthesized tools are available.Source: §5.4 · p. 4
• Bash+Synthesis packages code into shorter invocations; TheAgentCompany uses more static utility commands, while APEX-Agents uses longer embedded programs.The shell therefore supports both utility-based workflows and file analysis.Source: §5.4 · p. 5, Figure 5 · p. 5

5.5 Tool Synthesis

• Reuse does not consistently improve call success: rates are similar for Opus-4.8 but worse for GPT-5.5, and invoking synthesized tools yields no detectable score gain over Bash.Most created tools are invoked, but fewer reach later tasks.Source: §5.5 · pp. 5–6

5.6 Programmatic Tool Calling

• PTC substantially reduces uncached input and cache-read tokens relative to Tool-only on both benchmarks, with broadly similar output-token use.The authors suggest that processing intermediate results within programs and returning selected outputs may reduce context carried into later model requests.Source: §5.6 · p. 6

6 Discussion

• Shell-only agents offer the strongest overall task performance and cost efficiency, while PTC underperforms shell-only on both quality and cost.The discussion recommends shell-only where arbitrary execution can be isolated and PTC when security or compliance restricts actions to an approved catalog.Source: §6 · p. 6
• Adding a typed catalog or persistent synthesized-tool library to the shell provides no consistent quality gain.Source: §6 · p. 6
• Opus-4.8 is suggested for shell-only coding and cross-application work, while GPT-5.5 is suggested for token-efficient direct tool calls.For document-heavy analysis and PTC, the discussion frames higher quality with Opus-4.8 against lower token use with GPT-5.5.Source: §6 · p. 6
• Platforms could expose direct typed calls, shell-only execution, and catalog-restricted programmatic orchestration as configurable options.Practitioners could select based on task needs and whether general code execution is permitted.Source: §6 · p. 6
• A/B tests on workplace scenarios could compare user-need fit, manual revision, operating cost, and operational failures with permissions and human review held constant.These evaluations could help teams choose the simplest interface meeting quality, cost, and permission requirements.Source: §6 · p. 6

A Harness and Prompts

• Prompts share benchmark identity and task text across interfaces, while synthesis and PTC add interface-specific guidance and synthesis encourages reusable parameterized commands.The benchmark setup overrides the GitHub Copilot SDK’s default toolset and system prompts; the SDK supplies the agent loop.Source: A Harness and Prompts · p. 8

B.1 TheAgentCompany

• TheAgentCompany combines cross-application workflows with local coding and single-application tasks, including resume screening across ownCloud and RocketChat.Another example synchronizes Plane issues with GitLab.Source: §B.1 · p. 8
• Its typed catalog covers task-relevant service APIs and workspace operations, but not every shell-accessible endpoint.With no canonical benchmark system prompt, the authors use a shared identity and vary tools and interface-specific guidance.Source: §B.1 · p. 8
• Uniform coworker patches recover missed channel messages on the next turn, prevent premature thread closure, and reset Redis between tasks to prevent memory carryover.GPT-5 coworkers remain fixed across models and interfaces.Source: §B.1 · p. 8
• Reported TheAgentCompany runs mitigate shell-enabled reward-hacking shortcuts by blocking access to the public evaluator repository and encrypting coworker information.Observed shortcuts included reading evaluator scripts or simulated-coworker files instead of completing intended workflows or contacting coworkers.Source: §B.1 · p. 8

B.2 APEX-Agents

• APEX-Agents spans 33 scenario worlds across investment banking, management consulting, and corporate law, with equal task counts across domains.Tasks include contract review, spreadsheet valuation, and survey reconciliation; most request written answers, while others require document, spreadsheet, or presentation edits.Source: §B.2 · p. 8
• The APEX snapshot combines application servers with a Unix sh -c server, enforcing filesystem restrictions, environment-variable scrubbing, and execution limits.The catalog revision predates FMP and EDGAR SEC integrations, avoiding FMP’s paid external-API-key online mode.Source: §B.2 · p. 8
• An Archipelago-derived judge scores criteria from the task, final response, and workspace-file changes, and passing requires every criterion.Inference uses the authors’ harness; the public APEX leaderboard uses maximal reasoning and a later server snapshot, so absolute scores are not directly comparable.Source: §B.2 · p. 8

C.1 Scores by Domain

• Every domain follows the score ordering Bash ≥ Bash+Tool ≥ Tool-only, with the greatest separation in software engineering and closely grouped interfaces in Corporate Law.Source: §C.1 · p. 8

C.2 Paired Score Differences

• Paired score differences are averaged over matched task-model pairs from both models, with each pair weighted equally.The catalog captions report 60 typed tools for TheAgentCompany and 20 tools across 8 non-shell servers for APEX-Agents.Source: §C.2 · p. 8, Table 7 · p. 9, Table 8 · p. 9

C.3 Shell Execution Details

• On APEX-Agents, GPT-5.5 uses more heredoc markers and fewer logical-chain markers than Opus-4.8 under Bash, Bash+Tool, and Bash+Synthesis.Source: §C.3 · p. 9
• The Repair metric counts a retry only when the next shell call after failure shares the first command, missing returns after intervening steps that do not confirm repair.Logged sequences such as python → pwd → python show a return to the executable, not a confirmed repair or response to the failure.Source: §C.3 · p. 9

• Some synthesized tools lack direct typed-tool counterparts, including financial-modeling scripts and general-purpose extract_text.py.Source: §C.4 · p. 9
• Reuse tracking counts overwritten paths once, requires creation and invocation in an earlier task for cross-task reuse, and excludes write/update calls.Transient or deleted tools may be missed.Source: §C.4 · p. 9
• For GPT-5.5 on APEX-Agents, reuse calls face more argument rejections; excluding them leaves only a small success-rate difference from fresh inline code.The check retains unknown outcomes and measures whole-call status, not individual tool-body reliability.Source: §C.4 · pp. 9–10

• An LLM judge assigns each failed trajectory to one failure category.Source: §C.5 · p. 10
• TheAgentCompany failures span incomplete deliverables, skipped verification, guessed values, wrong retrievals or deliveries, and environment/access issues.Source: §C.5 · p. 10
• In TheAgentCompany, tool-only failures most often involve incomplete deliverables or environment/access, shell-based failures often omit verification, and PTC has the highest final-message truncation share.Source: §C.5 · p. 10
• Failure profiles differ more by benchmark than interface: APEX-Agents failures center on omitted requirements, misapplied domain rules, and incorrect numbers or formats.Source: §C.5 · p. 10, Figure 7 · p. 13