PaperlayerView exact revision on arXiv

Source-linked AI summary

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye

arXiv:2605.22535v1cs.AI

TL;DR

Reliable evaluation of agents on authentic terminal workflows remains unresolved because existing benchmarks can diverge from real-world practice and scale poorly. TerminalWorld reverse-engineers validated tasks from in-the-wild recordings, and benchmarking shows that even the best system solves only 62.5% of verified tasks.

  • Problem

    Reliable evaluation of agents on authentic real-world terminal tasks remains an open question because existing benchmarks can diverge from workflows and scale poorly.

  • Method

    TerminalWorld automatically reverse-engineers executable, rigorously validated evaluation tasks from naturally occurring terminal recordings.

  • Results

    Current systems still struggle with real-world terminal workflows: even the best model solves only 62.5% of verified tasks.

  • Takeaways & Limitations

    Terminal-Bench scores are only weakly predictive of performance on TerminalWorld-Verified, indicating that existing expert-curated challenges do not fully capture real-world terminal capabilities.

  • Takeaways & Limitations

    The evaluation is scoped to pure CLI workflows, excluding TUI-based interactions.

Abstract

from arXiv · show

We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld.

1 Introduction

TerminalWorld reverse-engineers authentic evaluation tasks from voluntarily shared terminal recordings, addressing the realism and scalability limits of manually curated benchmarks. Its validated benchmark and manually reviewed subset expose persistent weaknesses in frontier models and weak alignment with Terminal-Bench performance.

  • Motivation: Terminal agents increasingly automate multi-step workflows by issuing commands, composing tools, and interpreting feedback in interactive CLI sessions.These capabilities are exemplified by frameworks including SWE-agent and OpenHands.
  • Problem: Manually curated benchmarks can diverge from authentic workflows because experts prioritize adversarial puzzles, while labor-intensive authoring scales poorly with evolving practices and tools.Terminal-Bench and LongCLI-Bench are cited examples of this benchmark approach.
  • Approach: TerminalWorld reverse-engineers naturally occurring terminal operations from structured asciinema recordings containing commands and corresponding system responses.The recordings provide a self-curated, human-vetted, continuously growing corpus of developer work.
  • Benchmark: 80,870 raw recordings yield 1,530 validated tasks spanning 18 categories, over 50-step workflows, and 1,280 unique commands, with 91% absent from Terminal-Bench.The automated pipeline can be rerun as asciinema accumulates new uploads, supporting continued scale.
  • Benchmark: 200 tasks form the VERIFIED subset, with authors manually executing reference solutions in reproduced environments and auditing semantic alignment across artifacts.The full benchmark represents in-the-wild terminal usage, while VERIFIED is intended as a rigorous testbed for frontier models and agents.
  • Findings: 62.5% is the best model pass rate, while Terminal-Bench scores weakly predict TERMINALWORLD-VERIFIED performance with Pearson r = 0.20.The analysis also reports that agent frameworks mainly influence cost-effectiveness rather than the underlying capability ceiling.

2 Related Work

Prior work has developed terminal agents that autonomously execute multi-step workflows through interactive command-line environments. Related benchmarks have expanded from isolated command-line skills toward end-to-end agentic assessment in interactive shell sandboxes.

  • Terminal Agents: Terminal agents issue commands, compose tools, execute multi-step workflows, and interpret execution feedback in interactive CLI environments.Early frameworks wrap shell commands behind structured tool APIs, constraining agent actions through explicit schemas.
  • Benchmarks for Terminal Agents: Terminal-agent benchmarks have progressed from narrow, isolated command-line abilities to complex end-to-end assessment in interactive shell sandboxes.Earlier examples include natural-language-to-shell translation and single-turn command execution with interactive feedback.

3 TERMINALWORLD: Scalable Data Engine for Real-World Terminal Tasks

TERMINALWORLD automatically reverse-engineers pure CLI tasks from high-fidelity human terminal recordings, then reconstructs executable environments and validates generated tests through execution feedback. Its pipeline emphasizes authenticity, deterministic outcomes, and filtering for reproducibility and safety.

  • Data Collection: 80,870 human terminal recordings are collected from asciinema, whose transcripts preserve ordered commands and standard outputs as high-fidelity execution logs.Metadata such as titles and descriptions is collected alongside each transcript.
  • Data Filtering: The pipeline filters recordings for privacy, safety, pure CLI scope, Docker reproducibility, and nontrivial length before task synthesis.It excludes PII, credentials, malicious commands, TUIs, GUI applications, inaccessible dependencies, Windows environments, proprietary software, and aborted or trivial sessions.
  • Task Synthesis: LLM-based synthesis converts noisy transcripts into outcome-oriented instructions and clean executable bash reference solutions.The instructions describe the expected final state rather than procedural steps, while extracted commands are deduplicated and assembled into coherent workflows.
  • Environment Reproduction: The engine synthesizes Docker environments by inferring dependencies from reference solutions and iteratively repairing build and execution failures with feedback.For linked repositories, the agent clones and scans projects to infer environment requirements; the loop diagnoses build errors and missing dependencies.
  • Test Validation: A task is admitted only when its generated test suite passes all three trials; otherwise, targeted repairs continue until success or the task is discarded.Instruction updates are synchronized with test-suite adjustments when needed, and unrepaired suites are discarded within the computational budget.

4 The TERMINALWORLD Benchmark

TERMINALWORLD is a self-curated, human-vetted benchmark derived from naturally occurring developer workflows, covering broad real-world terminal usage across diverse categories, complexities, and commands. Its VERIFIED subset provides a manually reviewed, rigorous testbed of 200 representative tasks for evaluating frontier models and agents.

  • Benchmark construction: The dataset is self-curated and human-vetted by construction because it derives from naturally occurring terminal recordings voluntarily uploaded by developers.This supports comprehensive coverage of everyday tools, configurations, and problem-solving strategies.
  • Coverage and diversity: 1,530 tasks span 18 real-world terminal categories, including underrepresented workflows such as container orchestration, CI/CD pipelines, and cloud infrastructure management.The benchmark is compared with 241 unique tasks deduplicated across Terminal-Bench 1.0 and 2.0.
  • Coverage and diversity: TERMINALWORLD covers a natural complexity spectrum from a few commands to workflows exceeding 50 steps, with dense representation of short everyday workflows.Its real-world derivation provides larger scale across all command-length intervals.
  • Coverage and diversity: 1,280 unique commands appear in reference solutions, and 91% are absent from Terminal-Bench across diverse environment-management and infrastructure-configuration tools.Examples include vagrant, virtualenv, terraform, and ansible.
  • VERIFIED subset: 200 representative tasks comprise the VERIFIED subset, balancing diversity and complexity while prioritizing longer command sequences and non-trivial domain-specific tools.Four experienced authors manually reviewed the subset by executing reference solutions in containerized environments and checking artifact alignment.
  • VERIFIED subset: The VERIFIED subset offers a rigorous and challenging testbed for benchmarking frontier models and agents on complex, real-world terminal tasks.It is designed as a snapshot of in-the-wild terminal usage.

5 Benchmarking Results on TERMINALWORLD-VERIFIED

TERMINALWORLD-VERIFIED remains difficult for frontier models, revealing efficiency, domain, and framework effects beyond raw capability. Its weak correlation with Terminal-Bench and low command overlap with human workflows indicate that it measures distinct real-world terminal behavior.

  • Model Performance: 49.0%–62.5% overall pass rates (54.8% average) show that frontier models continue to struggle, with Claude Opus 4.7 failing over one-third of tasks.Open-weight models such as Kimi K2.6 and GLM 5.1 are highly competitive.
  • Efficiency: Pearson r = −0.49 for turns and r = −0.62 for tokens show weak negative correlations between resource consumption and task success.GPT-5.5 and MiniMax M2.7 use substantially more tokens and turns than most peers while achieving lower pass rates.
  • Domain Performance: 87.5% average performance on environment setup contrasts with 28.1% on performance optimization, while no single model dominates across all domains.Software build and test also performs strongly at 78.1%, whereas scripting and automation and debugging and testing remain low at 39.1% and 39.3%.
  • Agent Frameworks: $0.51–$0.55 per pass is achieved by Terminus-2 and mini-SWE-agent at roughly $60 total cost with Claude Opus 4.7, despite comparable pass rates across frameworks.Frameworks affect cost-effectiveness more substantially than the model’s capability ceiling.
  • Benchmark Transfer: Pearson r = 0.20 between Terminal-Bench 2.0 and TERMINALWORLD-VERIFIED scores accompanies a 57.0%–82.7% versus 49.0%–62.5% score gap, showing weak transfer to authentic workflows.Model rankings are reshuffled across benchmarks, and only Claude Opus 4.7 performs strongly on both.
  • Human Workflow Comparison: 21.4% median Jaccard overlap shows that agents usually solve tasks through command paths different from those of the original human practitioners.For packet analysis, agents use tshark with Python instead of ettercap; the passages present this as an alternative strategy that reaches the same outcome.

6 Conclusion

TERMINALWORLD reverse-engineers high-fidelity terminal evaluation tasks from in-the-wild human recordings at scale. Its 200-task VERIFIED subset shows that current frontier systems still struggle with real-world terminal workflows, often using command paths that differ from human workflows.

  • Benchmark findings: 21.4% median command-set overlap was observed between agents and human workflows.Agents often reached the correct outcome through different command paths despite tasks being derived from human recordings.
  • Benchmark construction: 80,870 raw asciinema recordings were distilled into 1,530 validated terminal tasks.The benchmark construction engine reverse-engineers evaluation tasks from in-the-wild human recordings.
  • Benchmark construction: 200 tasks were additionally human-verified as the VERIFIED subset.This subset supports benchmarking of frontier LLMs and terminal agents.
  • Benchmark findings: Current systems still struggle with real-world terminal workflows on TERMINALWORLD-VERIFIED.The conclusion reports this finding from benchmarking frontier LLMs and terminal agents.

A Broader Impact and Ethical Considerations

TerminalWorld addresses ethical risks in sourcing public terminal recordings through consent-aware retrieval, privacy filtering, copyright protections, and author-led verification. Its release model avoids rehosting original recordings while directing traffic to asciinema and emphasizing the value of authentic interaction traces for AI evaluation.

  • Data Sourcing and Consent: TerminalWorld retrieves publicly listed asciinema transcripts through standard download mechanisms and follows robots.txt directives.The framework treats public recordings as material available for their intended educational and collaborative use.
  • Copyright, Rehosting, and the Right to be Forgotten: The benchmark does not distribute original human transcripts or native .cast files, releasing only synthesized evaluation artifacts.This design prevents unauthorized redistribution of copyrighted material.
  • Privacy and PII Mitigation: Before task synthesis, the automated pipeline filters recordings containing PII, sensitive credentials, or operations demonstrating malicious intent.Examples of sensitive credentials include API keys and AWS tokens.
  • Human Quality Verification: Authors and expert collaborating researchers manually verified the final VERIFIED subset without using uncompensated or under-compensated crowdsourced labor.The verification process was conducted entirely by the paper’s authors and expert collaborators.
  • Platform Mutual Benefit: By using direct hyperlinks instead of rehosted files, TerminalWorld directs community traffic to asciinema and highlights platforms preserving authentic interaction traces for AI evaluation.The paper frames this model as mutually beneficial rather than exploitative.

B Terminal Task Scope and Categories in TERMINALWORLD

TerminalWorld defines tasks as pure command-line workflows in which agents use shell commands and observe stdout/stderr, excluding full-screen TUI interactions. It organizes tasks into 18 real-world categories labeled by practitioners’ primary goals rather than incidental tools.

  • Task definition: TerminalWorld evaluates pure CLI workflows, where agents issue shell commands and use stdout/stderr to choose subsequent actions.Full-screen TUI applications are outside the benchmark’s operational task definition.
  • Task definition: Terminal-based editors and other TUI programs are discarded during data collection, leaving TUI-based interaction evaluation for future work.Examples of excluded editors include vim, nano, and emacs.
  • Categories: 18 real-world terminal task categories organize TerminalWorld according to the purposes of source recordings.Each task receives a single category label.
  • Categories: Category labels reflect each task’s primary goal—the final practitioner-desired state—rather than the incidental tools used to achieve it.A shell script for nightly database backups is categorized as Database Operations, not Scripting & Automation.

C Detailed Experimental Setup and Analysis · C.1 Benchmarking Large Language Models

The LLM benchmark standardizes model evaluation through Terminus-2 and fixed Harbor configurations, while distinguishing harness errors from agent failures. Experiments use isolated Docker environments and report both standard and error-adjusted success rates.

  • C.1 Benchmarking Large Language Models: All models are evaluated with Terminus-2, Harbor’s native scaffold, using provider-default reasoning settings to isolate model capability from agent-framework differences.Experiments ran on a CPU-based server executing the Harbor harness and Docker containers.
  • C.1 Benchmarking Large Language Models: Table 4 reports per-model pass, failure, timeout, error, resolved-rate, and resource-use statistics under Terminus-2.Timeout is a subset of Fail, Err counts harness failures before agent execution, and averages cover only tasks where the agent ran.
  • C.1 Benchmarking Large Language Models: Harbor runs four concurrent trials per model, reuses cached Docker images, deletes containers after trials, and injects network_mode: none for self-contained evaluation.No Docker network pool exhaustion was observed across the experiments.
  • C.1 Benchmarking Large Language Models: Harness errors occur when evaluation fails before agent execution, with shared causes across models because all trials use the same Terminus-2 scaffold.The two identified categories are tmux session initialization failures and container startup timeouts.
  • C.1 Benchmarking Large Language Models: Tmux initialization can abort attempts through a nondeterministic adapter race condition that sends keystrokes before session initialization completes.This harness bug is independent of both task and model.
  • C.1 Benchmarking Large Language Models: Table 5 reports the corresponding detailed per-agent statistics, using the same timeout, error, and average-calculation conventions.This table is identified as covering the agent benchmarking experiment.
  • C.1 Benchmarking Large Language Models: 2.5% to 5.5%: observed error rates range across models, and results report both pass / total and pass / (total −errors).The resolved rate isolates task-solving capability from harness artifacts; agent timeouts remain failures rather than errors.

C.2 Benchmarking Terminal Agents · C.3 TERMINALWORLD vs. Terminal-Bench

TerminalWorld agents are evaluated in isolated Docker containers through the Harbor harness, with explicit handling for concurrency, retries, and harness-level errors. Comparisons with Terminal-Bench use official model scores there, versus a unified Terminus-2 scaffold with default reasoning effort on TERMINALWORLD.

  • C.2 Benchmarking Terminal Agents: Harbor provisions a fresh Docker container per task, injects each agent, and collects execution traces.Terminus-2 is natively integrated and requires only a working bash shell; third-party agents must be installed inside each container.
  • C.2 Benchmarking Terminal Agents: CLI experiments ran with concurrency n = 4 for all agents to limit Docker resource pressure and API rate-limit risk.The setting was shared with Terminus-2 experiments where not otherwise specified.
  • C.2 Benchmarking Terminal Agents: Errored tasks were rerun after fixing agent-installation scripts, and merged results kept the most recent result per task.Retry outputs were combined with non-error results from the original run.
  • C.2 Benchmarking Terminal Agents: Errors are harness failures before task execution, distinct from task failures in which agents execute commands but fail verification.The reported error categories are tmux initialization failures, container startup timeouts, and third-party installation failures on incompatible base images.
  • C.2 Benchmarking Terminal Agents: 4.5–21.5% versus 3.5–5.5%: third-party agents had higher error rates than Terminus-2.Approximately 10% of TERMINALWORLD tasks use legacy base images with end-of-life repositories, making in-container toolchain installation fail; Terminus-2 is unaffected.
  • C.3 TERMINALWORLD vs. Terminal-Bench: Terminal-Bench comparisons use each model’s best available official self-reported score, which may reflect different scaffolds and reasoning configurations.Re-running all models under identical Terminal-Bench conditions was infeasible.
  • C.3 TERMINALWORLD vs. Terminal-Bench: TERMINALWORLD evaluates all models under a unified Terminus-2 scaffold with default reasoning effort.Table 6 records the scaffold and reasoning-effort settings used for the Terminal-Bench and TERMINALWORLD experiments in Figure 4.

C.4 Behavior Comparison between Agents and Humans

Agents and humans share only 21.4% median command-set overlap, while agent success is more strongly shaped by reference workflow length than human completion time. Agents also typically issue more commands than references, especially on failed runs where exploration becomes unproductive.

  • Agent–Human Comparison: 21.4% median Jaccard similarity measures command-set overlap between agents and humans.The comparison also examines how success varies with task complexity and how agent command counts relate to reference solution length.
  • Success and Procedural Complexity: Human completion time is noisier than reference command count because long-running but short-command tasks can be easier for agents than procedurally complex tasks.Compilation or model training may take humans longer without requiring agents to execute many commands.
  • Agent Command Count: Failed runs cluster further above the reference-command diagonal than successful runs, indicating unproductive exploration when agents cannot identify the correct solution path.Agents typically issue more commands than the reference solution in most runs, and reference command count does not tightly predict agent command count.
Loading 2605.22535v1…