PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, Ludwig Schmidt

arXiv:2601.11868v1cs.SEcs.AI

TL;DR

Existing benchmarks provide limited evidence about agents’ ability to perform difficult, realistic work in terminal environments. The paper introduces Terminal-Bench 2.0, a curated 89-task benchmark with containerized environments, human-written solutions, and outcome-based tests. Frontier models and agents resolve less than 65% of tasks, and the benchmark’s limitations include internet-enabled external dependencies and contamination risks.

  • Problem

    Existing benchmarks do not sufficiently measure realistic, difficult tasks needed to assess frontier agents’ capabilities.

  • Method

    The paper builds 89 manually verified terminal tasks with containerized environments, instructions, tests, reference solutions, and time limits.

  • Results

    Less than 65% of tasks were resolved by frontier models and agents on Terminal-Bench 2.0.

  • Takeaways & Limitations

    Terminal-Bench provides a challenging evaluation framework for measuring agentic LLM performance on realistic technical tasks.

  • Takeaways & Limitations

    Internet access introduces changing external dependencies, while developers could train on the dataset and intentional contamination remains difficult to prevent.

Abstract

from arXiv · show

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .

1 INTRODUCTION

Terminal-Bench 2.0 addresses the need for realistic, difficult benchmarks by evaluating agents on high-skill terminal tasks. Across its 89 tasks, frontier models and agents resolve less than 65% of tasks, while smaller models score around 15%.

  • Terminal-Bench evaluates agents on realistic command-line tasks requiring domain knowledge, long action chains, and autonomous problem-solving.
  • The framework covers professional work such as legacy-system configuration, research-paper reimplementation, and general software engineering.
  • 89 challenging tasks were manually verified by three human reviewers for correctness.
  • Less than 65% of tasks were resolved by frontier models and agents, while smaller models scored around 15%.

2 TERMINAL-BENCH

Terminal-Bench combines interactive containerized tasks with outcome-based tests and reference solutions, supporting diverse, long-horizon workflows. Its construction uses contributor-generated tasks, automated checks, adversarial testing, and extensive human review.

  • 2.1 TASK FORMULATION: Each task provides an instruction, Docker image, tests, example solution, and time limit inside an interactive container.
  • 2.1 TASK FORMULATION: Tests evaluate the final container state rather than the agent’s commands or console output, allowing multiple solution strategies.
  • 2.2 DATASET CONSTRUCTION: 89 tasks were selected from 229 open-source contributions using author difficulty assessments and review by three experienced human reviewers.
  • 2.3 VERIFICATION: Verification checks include oracle-solution execution, contributor checklists, language-model-assisted review, and adversarial exploit agents.
  • 2.3 VERIFICATION: One task requires almost 1 day for an expert and 10 days for a junior engineer, illustrating the framework’s long-horizon task range.
  • 2.4 COMPOSITION: Tasks span software engineering and nonengineering categories including personal assistant and video processing, with no single category forming a majority.

3 EXPERIMENTAL SETUP

The experiments evaluate six agents across 16 frontier models, using repeated trials and varied agent scaffolds. A neutral Bash-only testbed addresses the difficulty of separating model and agent performance.

  • 32,155 trials evaluate six agents across 16 frontier models, with each supported model-agent combination run at least five times.Each trial is one agent’s attempt to solve a single task.
  • Agent and model performance are hard to decouple because scaffolds often accommodate particular models, especially within the same organization.Agents also vary in their available tools and how they manipulate the container.
  • Terminus 2 provides a neutral comparison scaffold with a single headless terminal and Bash-only task completion.It was created to account for constraints and biases in other agents while preserving the benchmark’s terminal premise.
  • The evaluation covers Claude Code, Codex CLI, Gemini CLI, OpenHands, Mini-SWE-Agent, and Terminus 2.The agents include three command-line agents and three open-source software engineering agents.
  • Models include top closed-source and popular open-weight systems, accessed through first-party or Together.AI APIs respectively.Each model is evaluated using its compatible agent scaffold.
  • Terminal-Bench 2.0 tasks are implemented in Harbor and executed with its evaluation harness, using 32–100 Daytona containers in parallel.The benchmark is distributed through the Harbor registry.

4 RESULTS

Terminal-Bench 2.0 exposes substantial performance gaps across frontier agents, with results varying by model, cost, release date, and task difficulty. Error analysis finds execution failures dominant and distinct failure profiles across models.

  • Model performance: 52% is the Codex CLI resolution-rate increase when using GPT-5.2 instead of GPT-5-Nano.Gemini-2.5-Pro’s resolution rate increases by 17% when paired with Terminus 2 instead of OpenHands.
  • Cost and model performance: 1–100 dollars is the reported cost range for running Terminal-Bench 2.0, while some tasks take up to two hours and almost 100 million tokens.Average turns and token count do not necessarily correlate with better performance.
  • Model performance over time: Nearly doubled is the state-of-the-art performance increase between Gemini 2.5 Pro and GPT-5.2 over eight months.The paper reports improved performance with new model releases and anticipates possible future automation of well-defined terminal tasks.
  • Predicted vs. empirical difficulty: 93.3% of human-hard tasks are also empirically hard, with human-predicted and empirical difficulty positively correlated at r = 0.436, p < 0.001.Empirical difficulty is based on Terminus 2’s average pass rate across selected frontier models, using thresholds of ≥66.7%, 33.3%–66.7%, and <33.3%.
  • Trajectory-level error analysis: Execution errors dominate the evaluated failure modes, while coherence and verification errors occur at lower and comparable rates.The frontier closed-source models show similar error profiles, whereas Qwen Coder has a more balanced pattern across failure modes.

5 LIMITATIONS

Terminal-Bench 2.0 balances realism and diversity against risks to reproducibility, verification, contamination control, and future benchmark saturation.

  • Dataset contamination remains difficult to prevent because model developers could train on the released benchmark, despite repository canary strings.The authors consider a private test set outside the paper’s scope because building one would require substantial community investment.
  • Internet access supports realistic package installation and web queries but introduces changing external dependencies and variable execution environments.Agents may call APIs or download packages, while machine resources and container runtime enforcement can differ.
  • Crowdsourced task diversity and realism are retained despite the possibility that some tasks may fail to meet verification criteria.The authors describe easier verification without crowd contributions as an unacceptable loss of diversity and realism.

6 RELATED WORK

Related benchmarks target software engineering, tool use, computer use, or narrow command-line skills, whereas Terminal-Bench evaluates general agentic computer manipulation through realistic terminal tasks.

  • Existing benchmark families cover software engineering, tool use, and computer use across tasks such as coding, function calling, and web interaction.
  • Other command-line benchmarks isolate shell-script optimization, software-environment configuration, or natural-language-to-Bash translation.
  • Terminal-Bench focuses on general agentic manipulation of computers rather than a single command-line capability.

7 CONCLUSION

Terminal-Bench introduces a challenging evaluation framework and dataset built from real-world tasks, finding that validating task correctness is difficult and that harder benchmarks are needed as LLMs improve.

  • Terminal-Bench provides an evaluation framework and Terminal-Bench 2.0 dataset for assessing agentic LLMs on challenging tasks.
  • Real-world tasks offer one route toward increasingly challenging datasets as LLM capabilities grow.
  • The benchmark’s results are reported for every agent and model combination in Table 2.

A.2 AGENT RUN ANALYSIS

The agent-run analysis records execution behavior, task-level outcomes, reproducibility checks, and failure patterns to assess both benchmark performance and task quality.

  • Run statistics: 32,155 trials record agent execution time and token usage, while Terminus 2 runs also record model API calls.
  • Run statistics: Most trials finish under 20 minutes, use fewer than 25 model calls, and consume under 10 million tokens, but extreme runs reach two hours and nearly 100 million tokens.
  • Per-task results: Task-level results are averaged over all trials for each task-agent-model combination across performance, timeout, and token-usage analyses.
  • Quality control: Quality control combines contributor checks, deterministic CI checks, LLM-backed checks, adversarial auditing, and human inspection before task inclusion.
  • Quality control: The benchmark checks task artifacts for reproducibility, explicit specifications, informative tests, anti-cheating measures, and required filenames.
  • Quality control: An LLM-backed quality process removes common task problems before human review, while a debug tool distinguishes specification failures from normal agent limitations.
  • Failure analysis: For multiple trials, consistent failures across trials suggest instruction problems, whereas differing failures may indicate fair task difficulty.
  • Failure analysis: An adversarial exploit agent tests whether tasks can be passed through shortcuts such as monkey-patching, guessing, or generating possible answers.

B.5 ASSESSMENT OF TERMINAL-BENCH ON THE AGENTIC BENCHMARK CHECKLIST (ABC).

Terminal-Bench performs strongly on the ABC checklist overall, with full task validity but documented weaknesses in objective test-case metrics, reproducibility, contamination prevention, and flaw-impact analysis.

  • Outcome validity: Test-case quality lacks objective automated metrics and instead relies on extensive human review of specifications and coverage.
  • Outcome validity: Some nondeterminism remains because outcomes can vary with hardware, external API stability, and infrastructure reliability.Containerization, repeated runs, and confidence intervals mitigate but do not eliminate these reproducibility issues.
  • Task validity: Task validity is judged strong because independent inspection found the released tasks well-specified and largely unambiguous.
  • Benchmark reporting: Public task hosting and a canary string provide limited protection against dataset contamination, while flaw-impact effects are not quantitatively estimated.The checklist notes that a private held-out test set is outside the paper’s scope and that benchmark flaws lack quantitative impact estimates.
  • Overall assessment: 0.896 average ABC score places Terminal-Bench second among existing benchmarks.The component scores are 0.857 for Outcome Validity, 1.000 for Task Validity, and 0.830 for Benchmark Reporting.

C TRACE FAILURE DESCRIPTION AND EXAMPLES

Terminal-Bench develops a tailored taxonomy for single-agent terminal failures by adapting MAST and defining precise, rubric-based categories and decision procedures.

  • Taxonomy design: The authors adapt MAST by removing categories that do not apply to single-agent CLI systems.
  • Taxonomy design: The taxonomy organizes terminal-agent failures into Execution, Coherence, and Verification classes.
  • Rubrics: Separate rubrics for each failure mode provide precise definitions, decision boundaries, and mode-specific exclusion criteria.
  • Failure modes: Disobey Task Specification captures material contradictions of hard or soft directives, including wrong methods, sources of truth, constraints, or output locations.Fully corrected violations and response-format-only issues are excluded.
  • Failure modes: Step Repetition is matched when at least two semantically or conceptually identical actions occur within one phase across distinct blocks.Material strategy changes and the first retry after an incomplete attempt are excluded, while up to two performance-tuning reruns are allowed.
  • Failure modes: Unaware of Termination Conditions covers continuing after success or futility and declaring completion before success conditions are met.One verification turn after success is allowed, but continued identical failures or premature completion can trigger a match.

D ADAPTERS

Adapters integrate external benchmarks into Terminal-Bench by translating their task formats into a shared schema, validating parity, and enabling evaluation through one interface.

  • Adapter architecture: Adapters translate diverse benchmark formats into Terminal-Bench’s standardized schema for direct evaluation with supported agents.This avoids separate agent configurations while preserving the original benchmarks’ integrity.
  • Integration requirements: Adapter integration requires natural-language task descriptions, containerized environments, programmatic evaluation criteria, and optionally reference solutions.
  • Coverage: 26 adapters currently span software engineering and scientific-computing domains.The unified interface supports evaluation of any agent on any adapted benchmark and enables direct comparisons across previously incompatible benchmarks.

E ERRORS ANALYSIS - COMMAND FAILURES

Terminal-Bench’s command-failure analysis uses an LLM judge and a detailed taxonomy, while performance analysis finds that interaction-round counts do not predict success.

  • Analysis method: An LLM-as-judge identifies individual command failures across trajectories and classifies them using the failure-mode taxonomy.
  • Episode efficiency: r=-0.028, p=0.916 indicates negligible correlation between episode count and success rate.The analysis therefore reports that interaction-round counts do not predict task success.
  • Episode efficiency: 44% success with 10 episodes makes GPT-5 Codex the leading example, while GPT-5 reaches 35% success with 7 episodes.Claude Sonnet 4.5 records 43% success with 22 episodes.
  • Episode efficiency: Qwen 3 Coder 480B and GLM 4.5 Air use about 35 episodes yet achieve only 24% success.

G.2 OUTPUT TOKEN GENERATION VS. SUCCESS RATE

Output verbosity does not significantly predict task success: models producing more text are not consistently more effective, while some efficient models achieve high success rates.

  • The weak negative correlation between output tokens and success rate was not significant.The reported correlation is r=-0.170 with p=0.515.
  • Claude Sonnet 4.5 and Claude Opus 4.1 achieved top-tier success rates with relatively moderate token usage.Their success rates were 43% and 38%, respectively.
  • GPT-5-Nano generated approximately 60k average tokens but achieved only 8% success.The result is presented as an example of verbosity reflecting inefficient reasoning rather than thoroughness.
  • Effective models demonstrated efficiency in both token generation and episode utilization rather than relying on greater output volume or more turns.The findings motivate emphasis on decision quality and strategic reasoning in model evaluation and development.

H LIST OF TASKS IN TERMINAL-BENCH 2.0

Terminal-Bench 2.0 spans diverse professional tasks, including scientific computing, machine learning, security, debugging, system administration, and software engineering.

  • Terminal-Bench 2.0 includes tasks across software engineering, system administration, scientific computing, machine learning, security, debugging, data science, games, and personal assistance.The listed tasks range from circuit design and compiler builds to calendar scheduling and chess analysis.
  • Scientific Computing: Scientific-computing tasks include adaptive rejection sampling and Bayesian-network structure recovery followed by intervention and sample generation.The Bayesian-network task requires recovering a six-edge DAG, setting Y=0.0, and generating 10k samples.
  • Software Engineering and Debugging: Software-engineering and debugging tasks require exact functional behavior, such as COBOL-to-Python equivalence, circuit computation, package compatibility, and memory-safe fixes.Validation includes byte-for-byte output equivalence, compilation, test execution, and Valgrind checks.
  • System Administration: System-administration tasks include building legacy software from source and configuring Git-to-web-server synchronization.Examples include CompCert, pMARS, POV-Ray, and a Git server serving pushed content over HTTP.
  • Security: Security tasks include cracking a password-protected 7z archive and bypassing an HTML JavaScript filter to trigger alert().The latter task requires creating an HTML file that executes JavaScript after sanitization without user interaction.

I LLM USAGE

The paper documents model usage and benchmark evaluation artifacts, including task-level descriptions, trial-result tables, benchmark checklists, failure-mode mappings, and adapter listings.

  • Claude 4 Opus transformed author-provided task instructions into abbreviated descriptions in Appendix H.
  • GPT-5.2 with Codex CLI achieved the highest listed resolution rate at 62.9% ± 3.0%.Claude Opus 4.5 with Terminus 2 followed at 57.8% ± 2.5%.
  • Table 2 reports resolution rates with 95% confidence intervals and token counts for running all 74 Terminal-Bench 2.0 tasks.
  • The appendix includes the Agentic Benchmark Checklist and a mapping from original MAST failure modes to a merged CLI-agent taxonomy.
  • Table 5 lists 26 benchmarks integrated through adapters into Terminal-Bench format.Adapters support unified evaluation across previously separate benchmark task formats.
Loading 2601.11868v1…