PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, Ludwig Schmidt

arXiv:2601.11868v1cs.SEcs.AI

TL;DR

Existing benchmarks may not capture the diversity and difficulty of realistic frontier tasks, limiting evaluation of agents on valuable long-horizon work. Terminal-Bench 2.0 addresses this with 89 challenging, tested terminal tasks and finds that frontier models and agents resolve less than 65% of them.

  • Problem

    Existing benchmarks do not consistently measure real-world tasks at the diversity and difficulty needed to evaluate frontier agents.

    Source: §1 · p. 2

  • Method

    Terminal-Bench 2.0 provides 89 manually verified terminal tasks with containerized environments, instructions, reference solutions, and tests that evaluate outcomes.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Frontier models and agents resolve less than 65% of Terminal-Bench 2.0 tasks, while smaller models score around 15%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Terminal-Bench 2.0 offers a challenging evaluation framework for measuring agent performance on realistic command-line tasks.

    Source: §1 · p. 2, §7 · p. 11

    Internet access introduces external dependencies that can change over time, and differences in machine resources or container runtimes can alter task environments.

    Source: §5 · p. 10

Abstract

from arXiv · show

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .

1 INTRODUCTION

Terminal-Bench evaluates AI agents on realistic, high-skill command-line tasks through containerized environments, instructions, tests, and reference solutions. Its 89-task Terminal-Bench 2.0 benchmark finds that frontier models and agents resolve less than 65% of tasks, while smaller models score around 15%.

Introduction

  • As agents increasingly operate autonomously over long horizons and in valuable or high-risk domains, benchmarks must capture the diversity and difficulty of frontier tasks in realistic environments.The introduction motivates the terminal as a ubiquitous interface for software engineering, scientific computing, cybersecurity, and machine learning.Source: §1 · p. 2
  • Terminal-Bench 2.0 contains 89 challenging tasks manually verified by three human reviewers, requiring domain knowledge, long action chains, and autonomous problem-solving.The tasks cover configuring legacy systems, reimplementing research papers, and general software engineering.Source: §1 · p. 2
  • Each task combines a containerized environment with relevant packages and files, an instruction, completion tests, and a manually written reference solution.This framework evaluates whether agents can perform high-skill work that professionals are paid to do in command-line interfaces.Source: §1 · p. 2
  • Frontier models and agents resolve less than 65% of Terminal-Bench 2.0 tasks, while smaller models score around 15%.The benchmark comprises 89 tasks, and the paper presents a taxonomy of failure modes to guide future model and agent development.Source: §1 · p. 2

2 TERMINAL-BENCH

Terminal-Bench 2.0 is an outcome-driven framework for interactive terminal tasks, built as a diverse 89-task benchmark through crowdsourced construction and extensive verification. Its tasks support varied, complex workflows across engineering and nonengineering domains.

2.1 TASK FORMULATION

  • Each task pairs an instruction, Docker image, tests, example solution, and time limit, with tests evaluating only the final container state rather than the agent’s commands.This outcome-driven design allows agents to use different approaches while completing interactive tasks through tools such as file editing and Bash commands.Source: §2.1 · p. 2

2.2 DATASET CONSTRUCTION

  • 93 contributors created 229 candidate tasks, from which 89 were selected for Terminal-Bench 2.0 using author difficulty assessments and review by three experienced human reviewers.The framework’s flexibility and expressiveness support diverse task designs, with detailed task descriptions provided in Appendix H.Source: §2.2 · p. 3

2.3 VERIFICATION

  • Verification required tasks to be specific, solvable, and resistant to real-world shortcuts, with oracle solutions confirming test-passing workflows and audits checking for design flaws and cheating paths.The process included automated checks, contributor checklists, language-model-assisted inspection, repeated model and Terminus 2 runs, adversarial exploit testing, and additional manual audits; the final benchmark received roughly three reviewer-hours per task.Source: §2.3 · pp. 4–5

2.4 COMPOSITION

  • Tasks span substantially different workflows, including asynchronous Python job management and COBOL-to-Python rewrites, while covering categories led by software engineering but extending to personal assistance and video processing.Authors also estimated completion times for junior engineers and domain experts, capturing the benchmark’s capacity for complex, long-horizon tasks.Source: §2.4 · p. 5, Figure 4 · p. 5, Table 1 · p. 5

3 EXPERIMENTAL SETUP

The experiments evaluate six agents across 16 frontier models, repeating compatible agent–model configurations for 32,155 trials. The setup also introduces Terminus 2 as a neutral Bash-only testbed and uses Harbor for scalable containerized execution.

3 EXPERIMENTAL SETUP

  • 32,155 trials evaluate six agents across 16 frontier models, with each supported agent–model combination run at least five times.Each trial is one agent’s attempt to solve a single task.Source: §3 · p. 6

3.1 TERMINUS 2

  • Terminus 2 provides a neutral comparison scaffold with one headless-terminal tool and Bash-only task execution, addressing agent–model coupling and tool-use variability.Agents may otherwise be tailored to particular models and use executable tools beyond ordinary Bash commands.Source: §3.1 · p. 6

3.2 AGENTS

  • The evaluation covers Claude Code, Codex CLI, Gemini CLI, OpenHands (Wang et al., 2025), Mini-SWE-Agent (Yang et al., 2024), and Terminus 2.Source: §3.2 · p. 6

3.3 MODELS

  • The model suite combines top closed-source and popular open-weight models, using provider-default reasoning effort and compatible agent scaffolds for each model.Closed-source models use first-party APIs, while open-weight models use the Together.AI API; company-specific CLIs run with their respective models.Source: §3.3 · p. 6

3.4 HARBOR

  • Harbor implements and distributes Terminal-Bench 2.0 evaluations, which run with Daytona using 32–100 containers in parallel.The benchmark is represented in Harbor’s task format and executed through its evaluation harness.Source: §3.4 · p. 6

4 RESULTS

Terminal-Bench 2.0 reveals substantial performance variation across models and agents, with GPT-5.2 paired with Codex CLI achieving the highest average resolution rate. Results also characterize costs, performance trends, task difficulty, and recurring trajectory- and command-level failures.

4 RESULTS

  • 63% is the highest average resolution rate, achieved by GPT-5.2 with Codex CLI, followed by Claude Opus 4.5 and Gemini 3 Pro at 58% and 57%.Proprietary models occupy the top 13 positions, while Terminus 2 and Kimi K2 Thinking perform best among open-weight models at 36%; model selection generally matters more than agent scaffold.Source: §4 · pp. 6–7, Figure 5 · p. 7

4.1 COST AND MODEL PERFORMANCE

  • $1–$100 is the cost range for running Terminal-Bench 2.0, while some tasks take up to two hours, use nearly 100 million tokens, and show no correlation between turns or token count and success.Most agents attempt tasks in under 20 minutes, but interaction patterns vary widely.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 MODEL PERFORMANCE OVER TIME

  • Nearly doubling state-of-the-art performance occurred between Gemini 2.5 Pro and GPT-5.2 releases over eight months, although the benchmark may become saturated within a year.The authors plan continued evaluations and new challenging task sets as model capabilities improve.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3 PREDICTED VS. EMPIRICAL DIFFICULTY

  • 93.3% of human-hard tasks are empirically hard, with human-predicted and empirical difficulty positively correlated at r = 0.436, p < 0.001.Empirical difficulty uses Terminus 2’s pass rates across frontier models, classifying tasks as easy, medium, or hard at thresholds of ≥66.7%, 33.3%–66.7%, and <33.3%. Medium-rated tasks are often harder for models, with 54.5% empirically hard.Source: §4.3 · pp. 8–9, Figure 7 · p. 8

4.4 TRAJECTORY-LEVEL ERROR ANALYSIS

  • Execution errors dominate Opus 4.5 and GPT-5.2 failures, whereas Qwen Coder shows a more balanced pattern across execution, coherence, and verification errors.The analysis uses two failed Terminus 2 trials per model; annotators achieved 93% Cohen’s-κ calibration agreement and 90% agreement with human annotations.Source: §4.4 · p. 9, Figure 8 · p. 9

4.5 COMMAND-LEVEL ERROR ANALYSIS

  • 24.1% of sampled command failures involve executables that are missing or absent from PATH, followed by executable-running failures at 9.6%.Across models, command error rates range from 9.2% for Grok 4 to 26.7% for GPT-OSS-120B, based on LLM-as-judge analysis.Source: §4.5 · pp. 9–10, Figure 9 · p. 10

5 LIMITATIONS

Terminal-Bench’s realism introduces risks from internet access, dataset contamination, and changing execution environments. Despite extensive review, some tasks may not fully meet verification criteria, reflecting a tradeoff between benchmark diversity and ease of verification.

Risks from access and contamination

  • Internet access could enable agents to find oracle solutions and cheat, although this behavior has not appeared in tens of thousands of trajectories.Benchmark users should remain vigilant.Source: §5 · p. 10
  • Developers could train on Terminal-Bench despite repository canary strings, while preventing intentional contamination would require a private test set beyond this paper’s scope.The authors identify private test sets as future work because creating one would require substantial community investment.Source: §5 · p. 10

Reproducibility limits

  • Internet calls, downloaded packages, changing resources, machine variability, and container-runtime enforcement can undermine otherwise reproducible task environments.The benchmark pins package versions, provides prebuilt Docker images, and requires dependencies in the Docker context, but external dependencies remain.Source: §5 · p. 10

Verification limits

  • Manual and language-model-assisted review cannot guarantee that every task meets verification criteria, but the authors accept this risk to preserve diversity and realism.A less diverse benchmark without crowd contributions would be easier to verify, and the authors invite community requests for task changes.Source: §5 · pp. 10–11

6 RELATED WORK

Terminal-Bench complements benchmarks for software engineering, tool use, computer use, and scientific discovery by evaluating diverse, expert-collected, long-horizon tasks in a real terminal shell. Unlike narrower command-line benchmarks, it targets general agentic computer manipulation.

Related Benchmarks

  • Terminal-Bench emphasizes diverse, long-horizon tasks collected from experts and conducted in a real terminal shell, distinguishing it from synthetic benchmark environments.This design targets tasks that are hard enough for frontier models, realistic, and economically valuable.Source: §6 · p. 11

Language Models and the Terminal

  • Unlike benchmarks targeting shell-script optimization, software-environment configuration, or natural-language-to-Bash translation, Terminal-Bench evaluates general agentic manipulation of computers.Source: §6 · p. 11

7 CONCLUSION · A DETAILED RESULTS · A.1 COMPREHENSIVE RESULTS

Terminal-Bench introduces a challenging evaluation framework and dataset for agentic LLMs, while emphasizing rigorous manual verification of task correctness. Comprehensive results for every agent–model combination are reported in Table 2.

7 CONCLUSION

  • Terminal-Bench introduces an evaluation framework and the Terminal-Bench 2.0 dataset for evaluating agentic LLMs on challenging tasks.Source: §7 · p. 11
  • The framework is motivated by the need to benchmark increasingly capable LLMs on increasingly challenging datasets.Source: §7 · p. 11
  • Terminal-Bench uses real-world tasks previously encountered by developers and researchers as a source of challenging agentic environments.Source: §7 · p. 11
  • Validating task correctness is extremely difficult, requiring substantial manual verification during dataset construction.The verification effort is intended to make cheating difficult and ensure that tested properties are adequately specified in task descriptions.Source: §7 · p. 11
  • Dataset curators are urged to invest significant manual effort in verifying challenging agentic environments.The recommendation covers both discouraging cheating and checking that task descriptions adequately specify all tested properties.Source: §7 · p. 11

A.1 COMPREHENSIVE RESULTS

  • Table 2 shows the results for every agent and model combination.Source: §A.1 · p. 21

A.2 AGENT RUN ANALYSIS … B.4 ADVERSARIAL EXPLOIT AGENT

The paper analyzes agent resource use and task-level performance, then describes layered quality-control procedures covering contributor checks, automated validation, debugging, and adversarial exploit testing.

A.2 AGENT RUN ANALYSIS

  • Most trials finished in under 20 minutes, used fewer than 25 model calls and 10 million tokens, while extremes reached 2 hours, hundreds of calls, and nearly 100 million tokens.Source: Figure 10 · p. 21, §A.2 · p. 21

A.3 TASK-LEVEL RESULTS

B QUALITY CONTROL

  • Quality control combines automated and manual verification, including a contributor checklist, deterministic and LLM-backed checks, debugging, and adversarial-agent auditing.Source: B QUALITY CONTROL · p. 21

B.1 TASK CONTRIBUTOR CHECKLIST

  • Contributors must verify tests and instructions match, document test behavior, prevent cheating, use human-written task artifacts, and pin reproducibility-sensitive dependencies.Source: §B.1 · pp. 21, 25

B.2 CHECK TOOLS

  • Deterministic checks reject missing canaries, embedded tests or solutions, unsafe privileges, invalid metadata, problematic file references, and dependency or packaging issues.Source: §B.2 · p. 25
  • LLM-backed checks assess specification-test alignment, documentation, anti-cheating protections, schemas, dependencies, typos, image contents, hardcoded solutions, and referenced output files, eliminating common problems before human review.Source: §B.2 · pp. 25–26

B.3 DEBUG TOOL

  • The debug tool distinguishes insufficient instructions from agent limitations by checking specification mismatches and repeated failure patterns across trials, returning a structured PASS, FAIL, or NOT_APPLICABLE judgment.Source: §B.3 · pp. 26–27

B.4 ADVERSARIAL EXPLOIT AGENT

  • An adversarial OpenHands agent attempts cheating without inspecting tests or oracle solutions; authors manually inspect trajectories for exploits such as monkey-patching, guessing, and incomplete-answer generation.Source: §B.4 · p. 27

B.5 ASSESSMENT OF TERMINAL-BENCH ON THE AGENTIC BENCHMARK CHECKLIST (ABC).

Terminal-Bench scores highly on the Agentic Benchmark Checklist, averaging 0.896 and ranking second among existing benchmarks. Its main weaknesses concern residual nondeterminism, public task availability enabling contamination, and the absence of quantitative flaw-impact estimates.

Outcome Validity

  • Outcome Validity is strong but lacks objective test-case quality metrics, and some nondeterminism remains from hardware, external APIs, and infrastructure reliability.Containerization, repeated runs, and confidence intervals mitigate reproducibility issues.Source: §B.5 · p. 27

Task Validity

  • Independent inspection found no major Task Validity issues, with released tasks well-specified and largely free of ambiguity or underspecification.Source: §B.5 · p. 28

Benchmark Reporting

  • Benchmark Reporting is limited because public GitHub tasks remain vulnerable to training contamination despite a symbolic canary, and the paper lacks quantitative estimates of flaw impacts.Source: §B.5 · p. 28

Overall

  • 0.896 average ABC score ranks Terminal-Bench second among existing benchmarks, with scores of 0.857 for Outcome Validity, 1.000 for Task Validity, and 0.830 for Benchmark Reporting.The authors attribute this performance to substantial developer care and rigor.Source: §B.5 · p. 28
  • ABC collaboration with benchmark developers surfaced and resolved issues through the public GitHub repository, helping improve evaluation rigor and benchmark quality.Source: §B.5 · p. 28

C TRACE FAILURE DESCRIPTION AND EXAMPLES … C.3 FAILURE TAXONOMY EXAMPLES

This section presents a Terminal Agent Taxonomy spanning Execution, Coherence, and Verification, with refined rubrics, matching rules, exclusions, and evidence-based thresholds. It also specifies LLM-judge checks and illustrates failure modes through terminal episodes, including failed retrieval followed by guessing and discrepancies between stated reasoning and executed actions.

C TRACE FAILURE DESCRIPTION AND EXAMPLES

  • The taxonomy maps applicable MAST categories into three classes—Execution, Coherence, and Verification—and uses separate rubrics for more precise boundaries and mode-specific exclusions.Single-agent CLI constraints exclude multi-agent failures, explicit role distinctions, and clarification requests.Source: C TRACE FAILURE DESCRIPTION AND EXAMPLES · p. 29

C.1 TERMINAL AGENT TAXONOMY

  • The Terminal Agent Taxonomy covers specification disobedience, repetition, termination failures, reasoning–action mismatch, context loss, derailment, and verification failures.Verification failures include no or incorrect verification and weak verification; premature termination is defined separately from unawareness of termination conditions.Source: §C.1 · p. 29

C.2 LLM JUDGE PROMPTS

  • Each failure-mode rubric combines high-level framing, a decision procedure, and exclusion criteria, with Docent used to refine ambiguities before final classification.The refinement process supports summarization, search, clustering, targeted intervention, and clarification of recurring patterns.Source: §C.2 · p. 29
  • The judge applies materiality and correction rules: format-only issues, harmless deviations, legitimate recovery, and fully reversed violations generally receive NO MATCH, while repeated or blocking contradictions receive MATCH.Examples include requiring at least two identical attempts for repetition, allowing one post-success verification turn, and counting reasoning–action mismatch when contradictions are repeated or block completion.Source: §C.2 · pp. 30, 31, 33
  • Verification judgments prioritize substantive checks on core properties of the final deliverable, distinguish weak verification from data fabrication, and accept authoritative evaluators when no additional essentials are required.Self-assertions do not count, failing core results ignored at completion trigger a match, and fabricated data must be relied on to satisfy a requirement, pass a check, or serve as the deliverable.Source: §C.2 · pp. 33–34
  • LLM-judge prompts assess explicit requirements, authoritative evaluator checks, implied prerequisites, coverage, decisiveness, mitigation, verification, and specified tie-breakers or exclusions.The procedure extracts essentials, compares coverage, and checks decisiveness and mitigation while accounting for evaluator rules, absent verification, brittleness, inconclusive evidence, verified constraints, and strong coverage.Source: §C.2 · p. 34

C.3 FAILURE TAXONOMY EXAMPLES

  • The mteb-leaderboard example disobeys specification by guessing a top Scandinavian MTEB embedding model after retrieval attempts failed, then writing the guess to `/app/result.txt` and marking the task complete.The episode identifies `jinaai/jina-embeddings-v3` as the model without successfully fetching the leaderboard and confirms only that the file contains the written value.Source: §C.3 · p. 34
  • The tune-mjcf example exhibits reasoning-action mismatch: after proposing valid Jacobian-only edits with correctness and speed checks, the episode executes sparse and dense variants through evaluation without a reported outcome.The stated plan targets a 60% runtime requirement with identical final state, resets the model, inserts each Jacobian option, diffs the file, and runs `python3 /app/eval.py`.Source: §C.3 · p. 35

D ADAPTERS

Adapters provide a standardized interface for integrating external benchmarks into Terminal-Bench, enabling agent-agnostic evaluation while preserving benchmark semantics through validation. The framework currently supports 26 adapters across domains, consolidating evaluation infrastructure for reproducible cross-benchmark comparisons.

Adapter architecture

  • Adapters translate diverse benchmark formats into Terminal-Bench’s standardized schema, allowing any supported agent to evaluate adapted benchmarks without modification.This interface reduces redundant benchmark-specific engineering while preserving the integrity of the original benchmarks.Source: Figure 14 · p. 36, D ADAPTERS · p. 36

Adapter development

  • Adapter integration requires natural-language tasks, reproducible containerized environments, programmatic evaluation criteria, and optionally reference solutions.Developers validate translations through parity experiments comparing identical agents and models on original and adapted benchmarks across multiple trials before integration.Source: D ADAPTERS · p. 36

Supported adapters

  • 26 adapters currently span domains from software engineering to scientific computing, enabling single-interface evaluation and direct comparison across previously incompatible benchmarks.The consolidated infrastructure supports reproducibility and systematic measurement of progress in agent development.Source: D ADAPTERS · p. 36

E ERRORS ANALYSIS - COMMAND FAILURES … G.1 EPISODE COUNT EFFICIENCY VS. SUCCESS RATE

This section examines terminal-agent command failures through a taxonomy and structured LLM-based classification of trajectory errors, including failure distributions across models and the minimal Terminus 2 agent. It also analyzes the relationship between episode count and success rate.

E ERRORS ANALYSIS - COMMAND FAILURES

E.2 COMMAND FAILURE MODE TAXONOMY

  • The taxonomy distinguishes failures spanning invocation, filesystems, environments, build tools, packages, networks, and runtime processes.Examples include missing executables or files, permission and configuration problems, dependency and toolchain failures, remote-access errors, crashes, and application-level failures.Source: §E.2 · pp. 46–48
  • The command-failure taxonomy distinguishes failures across pipelines, interpreters and REPLs, services and platforms, and data and formats, with specific resource, runtime, environment, parsing, and test categories.Examples include OOM, disk-full, file-descriptor, interpreter, unavailable-service, invalid-format, test-collection, assertion, performance, merge-conflict, and flaky-test failures.Source: §E.2 · pp. 49–50

E.3 SYSTEM PROMPT FOR FAILURE IDENTIFICATION

  • The failure-identification prompt asks the model to analyze terminal inputs and outputs, determine whether execution failed, and return the result as valid JSON with an `is failure present` Boolean.The prompt focuses on identifying failure indications in captured command outputs.Source: §E.3 · p. 50

E.4 SYSTEM PROMPT FOR TAXONOMY CLASSIFICATION

  • The taxonomy-classification prompt requires selecting the most specific applicable category and subcategory from the provided taxonomy and returning them as valid JSON fields.If no failure exists, both fields may be empty strings; category–subcategory pairs must come from the taxonomy.Source: §E.4 · pp. 50–51

F TERMINUS 2

  • Terminus 2 evaluates Terminal-Bench by having a language model issue keystrokes executed in a tmux shell inside a Docker container, then receiving the command output.Unlike agents with dedicated editing, Bash, or downloading tools, Terminus 2 operates solely through Bash commands and can use interactive shell capabilities.Source: F TERMINUS 2 · p. 51
  • Terminus 2 uses context summarization to condense context when the model reaches its limit, enabling extended tasks requiring millions of tokens.The summarization module makes calls to a language model to compress the agent’s context.Source: F TERMINUS 2 · p. 51

G MODEL PERFORMANCE: OUTPUT TOKENS AND EPISODE COUNT ANALYSIS

  • The performance analysis treats output-token generation and episode count as complementary metrics, defining an episode as one trial turn and averaging episode counts across each model’s Terminus 2 trials.Average episodes measure the interaction rounds typically required to resolve a task and support comparison with success rate.Source: G MODEL PERFORMANCE: OUTPUT TOKENS AND… · p. 51, §G.1 · p. 52

G.1 EPISODE COUNT EFFICIENCY VS. SUCCESS RATE

  • r=-0.028 (p=0.916) correlation shows that episode count does not predict task success; GPT-5 Codex leads at 44% success with 10 episodes per trial.Claude Sonnet 4.5 achieves 43% success with 22 episodes, while GPT-5 reaches 35% success with 7 episodes; Qwen 3 Coder 480B and GLM 4.5 Air use about 35 episodes but achieve 24% success.Source: §G.1 · p. 52, Figure 35 · p. 52

G.2 OUTPUT TOKEN GENERATION VS. SUCCESS RATE · G.3 IMPLICATIONS FOR MODEL EVALUATION

Across Terminus 2, output length was only weakly and nonsignificantly related to success, while strong performance could accompany moderate token use. The findings therefore emphasize efficient decision-making and strategic reasoning over verbosity or interaction volume.

G.2 OUTPUT TOKEN GENERATION VS. SUCCESS RATE

  • Average output tokens were collected for every trial and averaged across each model to measure verbosity and its relationship with task performance.The analysis tested whether generating more text correlates with better performance or whether conciseness can be equally effective.Source: §G.2 · p. 52
  • 43% and 38% success rates show that Claude Sonnet 4.5 and Claude Opus 4.1 achieve top-tier performance with relatively moderate token usage.These models demonstrate efficient problem-solving compared with the broader token-use pattern.Source: §G.2 · p. 52
  • r=-0.170, p=0.515 indicates a weak negative correlation between average output tokens and success rate, so verbosity does not significantly predict task success.Figure 36 reports the relationship across models in Terminus 2.Source: §G.2 · p. 52, Figure 36 · p. 53
  • 60k average tokens paired with 8% success for GPT-5-Nano indicates that excessive verbosity can reflect inefficient reasoning rather than thoroughness.GPT-5-Nano generates the most tokens by far among the contrasted models.Source: §G.2 · p. 53

G.3 IMPLICATIONS FOR MODEL EVALUATION

  • Verbosity and an extensive number of turns do not inherently lead to better performance in terminal environments.The evaluation implications distinguish interaction volume from effective model behavior.Source: §G.3 · p. 53
  • Future model development should prioritize decision quality and strategic reasoning rather than simply increasing output volume or interaction frequency.The most effective models demonstrate efficiency in both token generation and episode utilization.Source: §G.3 · p. 53

H LIST OF TASKS IN TERMINAL-BENCH 2.0

This section catalogs Terminal-Bench 2.0 tasks across diverse technical domains, including scientific computing, machine learning, security, systems, debugging, data processing, software engineering, modeling, video analysis, and file operations. The tasks require concrete, constrained, and testable artifacts validated through specified outputs, tests, performance constraints, or exact file artifacts.

Scientific Computing and Mathematics

  • The catalog includes scientific-computing tasks requiring statistical sampling, Bayesian-network intervention, DNA primer design, and cryptanalysis implementations with validation against distributions, causal samples, molecular constraints, or recovered keys.Examples include adaptive rejection sampling with formal tests, Bayesian-network intervention and resampling, Golden Gate and Q5 primer design, and FEAL differential or linear attacks.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 54, 59, 60, 61

Security

  • Security tasks cover both offensive and defensive workflows, including bypassing JavaScript filtering, cracking encrypted archives, removing XSS vectors, fixing CWE-class vulnerabilities, recovering leaked Git secrets, and generating validated TLS certificates.These tasks require producing exploit-triggering HTML, recovering archive contents, preserving safe HTML, documenting and repairing vulnerabilities, purging repository history, or validating certificate properties.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 54, 58, 61, 62, 63, 70

Software Engineering and Systems

  • Software-engineering and system-administration tasks span compiling legacy software, building simulators and interpreters, configuring Git and web infrastructure, and operating virtualized environments under explicit compatibility and deployment requirements.Requirements include NumPy-compatible Cython extensions, headless pMARS and POV-Ray builds, CompCert compilation, branch-specific Git deployment, Nginx services, and Windows 3.11 in QEMU with VNC access.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 55, 57, 63, 64

Machine Learning and Data Science

  • Machine-learning and data-science tasks require installing or serving models, training under fixed settings, constructing distributions or schedulers, extracting model parameters, and meeting quantitative accuracy, latency, or similarity criteria.Examples include CPU-only Caffe training for 500 iterations, a DistilBERT sentiment API, KL-constrained distributions, shape-aware LLM batching, RStan posterior estimation, and black-box ReLU weight extraction.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 55, 64, 66, 67, 68

File Operations and Data Processing

  • File-operation and data-processing tasks emphasize exact transformation and extraction, including WAL recovery, ELF parsing, video-command transcription, document classification, log aggregation, multi-source merging, and byte-for-byte COBOL modernization.Outputs are typically prescribed files such as JSON, CSV, Parquet, text, or rewritten data files, with requirements for completeness, accuracy, standardized formats, or exact equivalence.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 59, 60, 61, 66, 69

Software Engineering and Security

  • Tasks cover LaTeX warning removal, password recovery, image reconstruction, binary reverse engineering, polyglot Fibonacci programs, and Coq proof completion under strict file, size, compilation, or output constraints.Requirements include synonym-only edits, forensic password filtering, 0.99+ image similarity with sub-2KB source, identical binary behavior, dual-language compilation, and successful Coq compilation.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 70–71

Optimization and Scientific Computing

  • Optimization and scientific-computing tasks require accelerated portfolio calculations, constrained fusion-protein design, Raman peak fitting, and MuJoCo simulation speedups while preserving numerical or physical correctness.The portfolio extension must achieve at least 1.2x speedup with 1e-10 agreement; the protein design imposes sequence, linker, GC, and wavelength constraints; Raman outputs four parameters per peak; MuJoCo requires at least 40% faster simulation within atol=1e-5.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 72, 74, 78

Modeling, Video, and Program Construction

  • Modeling, vision, reverse-engineering, compression, and game tasks demand MNIST inference, selective PyTorch fine-tuning, fastText accuracy and size targets, video jump-frame detection, secret extraction, CoreWars win rates, and exact decompression.Required deliverables include executables, model artifacts, TOML or text outputs, a Redcode warrior meeting opponent-specific rates, and a compressed file no larger than 2500 bytes.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 73, 78, 79

Systems and Data Querying

  • Systems and data-query tasks configure QEMU SSH or telnet access, optimize an SQLite query without changing results, and construct regexes for chess moves and validated dates with edge-case handling.The tasks specify localhost ports and readiness behavior, SQLite-specific syntax, legal chess rules including castling and promotions, and date/IP boundary conditions.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 73–75

Data Science and Security

  • Data-science and security tasks reshard datasets reversibly, translate RStan inference to PyStan, refine histopathology masks with MobileSAM, and remove secrets from a repository’s history and files.The constraints include directory and file-size limits, PyStan 3.10.0 with seed 1, CPU-only unseen-data processing, and consistent credential placeholders without losing non-sensitive content.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 75–76

Programming, Databases, and Distributed Systems

  • Further tasks implement recursive Scheme evaluation, EU-university SPARQL filtering, truncated SQLite recovery, instrumented SQLite builds, and distributed pipeline or tensor parallelism.The parallelism tasks require correct communication, sharding, loss scaling, gradients, and support for specified devices, dtypes, and world sizes.Source: H LIST OF TASKS IN TERMINAL-BENCH 2.0 · pp. 76–78

I LLM USAGE

Terminal-Bench 2.0 reports agent-model trial results, with GPT-5.2 Codex CLI achieving the highest resolution rate at 62.9% ± 3.0%. Claude 4 Opus was also used to abbreviate author-provided task instructions for Appendix H.

LLM usage

  • Claude 4 Opus transformed the author-provided task instructions into the abbreviated “Description” in Appendix H.Source: I LLM USAGE · p. 80

Trial results

  • 62.9% ± 3.0% was the highest reported resolution rate, achieved by GPT-5.2 with Codex CLI, while Claude Sonnet 4.5 with Claude Code recorded 40.1% ± 2.9%.These trial results cover the evaluated agent-model combinations.Source: Table 2 · p. 81, I LLM USAGE · p. 81
  • Table 2 reports resolution rates with 95% confidence intervals and token counts for running all 74 Terminal-Bench 2.0 tasks.The table covers all evaluated agent-model combinations.Source: Table 2 · p. 81

Related papers

Loading 2601.11868v1…