Source-linked AI summary
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji, Feiyan Zhou, Xiaohui Zhang, Fanny Yang, Belinda Zeng
TL;DR
Existing benchmarks inadequately assess general-purpose terminal-use agents because they focus mainly on GUI interaction or programming-centric shell workflows. TUA-Bench addresses this gap with 120 curated tasks spanning everyday and expert workflows, finding that the strongest evaluated agent achieves only a 65.8% success rate.
Problem
Existing benchmarks focus mainly on GUI interaction or programming-centric terminal workflows, inadequately evaluating general-purpose terminal-use agents.
Method
TUA-Bench evaluates agents on 120 manually curated CLI tasks spanning everyday digital work and expert workflows, using executable environments and verification procedures.
Results
65.8% success rate is achieved by the strongest evaluated agent, indicating substantial gaps in terminal-based computer use.
Takeaways & Limitations
Reliable terminal-based computer use remains a challenging open problem across diverse digital and professional workflows.
Takeaways & Limitations
The benchmark covers only terminal-based workflows, a limited sample of specialized domains, and English-only task descriptions.
Abstract
from arXiv · showhide
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
1 Introduction
TUA-Bench addresses the limited coverage of existing terminal benchmarks by evaluating general-purpose agents across diverse everyday, scientific, and engineering workflows. Its 120-task benchmark and comprehensive evaluation show that frontier agents still face substantial challenges, with the strongest evaluated agent achieving 65.8% success.
- Motivation: Existing computer-use benchmarks primarily target GUIs, while terminal benchmarks mainly evaluate shell-native technical and programming workflows.This leaves general-purpose terminal-based computer use insufficiently evaluated.
- Benchmark: TUA-Bench contains 120 diverse CLI tasks organized into five families spanning everyday digital work and expert scientific and engineering workflows.The task distribution covers both routine computer activities and professional workflows co-designed with domain experts.
- Results: 65.8% success rate is achieved by the strongest evaluated agent, revealing substantial gaps in terminal long-horizon planning, tool use, monitoring, and error recovery.The evaluation includes frontier models, agent frameworks, and ablations on model reasoning effort.
2 Related Work
Prior benchmarks cover general computer use mainly through graphical interfaces or terminal workflows centered on software engineering and specialized command-line skills. TUA-Bench is positioned to address the open question of broad, professionally relevant computer operation through text-native terminal interfaces.
- GUI-based computer-use benchmarks: GUI benchmarks span browser tasks, live-web evaluation, and full desktop operation, assessing planning and error recovery alongside visual and spatial interaction skills.Examples include MiniWoB, WebVoyager, OSWorld, WindowsAgentArena, macOSWorld, and OSUniverse.
- GUI-based computer-use benchmarks: These GUI evaluations cover graphical computer use but leave open how effectively agents operate computers through text-native command interfaces.The unresolved issue concerns computer operation through command interfaces rather than graphical interaction.
- CLI-based agents and benchmarks: CLI agents use shell access to inspect codebases, edit files, execute tests, and iteratively resolve software tasks.SWE-agent and OpenHands exemplify this software-oriented use of command-line interaction, alongside deployed assistants such as Claude Code, Codex CLI, Gemini CLI, and OpenCode.
- CLI-based agents and benchmarks: Text-native benchmarks evaluate agents by commands, tool invocation, and verifiable outcomes, with Terminal-Bench targeting long-horizon tasks through executable tests and TerminalWorld using real-world terminal recordings.Other benchmarks address specialized command-line capabilities, including shell programming.
- Positioning of TUA-Bench: Existing benchmarks vary primarily by interaction interface and evaluated work scope, with broad or professionally relevant coverage generally relying on graphical or application-specific interfaces.OSWorld evaluates general desktop interaction through a GUI, while OfficeBench focuses on workflows across office applications.
3 TUA-Bench
TUA-Bench is a reproducible framework for evaluating terminal-use agents on broad everyday digital workflows and specialized scientific and engineering procedures. It contains 120 tasks organized into five families, with realistic environments, expert-designed workflows, and execution-based verification.
- Benchmark framework: TUA-Bench evaluates terminal agents on planning, realistic software interaction, file and artifact manipulation, and task-completion verification, not merely isolated command execution.The benchmark targets broad terminal-based capabilities across everyday and domain-specialized procedures.
- Execution infrastructure: Harbor orchestrates isolated, resettable container executions and collects trajectories, token usage, scores, and runtime metadata for reproducible evaluation.Each task is executed in a Linux container, with Docker and Podman supported by the infrastructure.
- Task specification and reproducibility: Every task packages instructions, artifacts, environment settings, runtime configuration, and an in-environment verifier into a self-contained specification.This packaging standardizes each task’s initial state, execution procedure, and evaluation protocol while preserving realistic variation.
- Task composition: 120 tasks span everyday work such as browsing, document and spreadsheet editing, email, and media processing, alongside expert workflows in biology, medical physics, architectural engineering, and mechanical engineering.The benchmark’s breadth and depth dimensions jointly cover general computer use and specialized professional procedures.
- Task curation: The everyday set adapts OSWorld tasks into command-line and file-level workflows, while professional tasks are co-designed with PhD-level experts and retain 20 challenging workflows after simplification filtering.Professional tasks use specialized tools and combine automatically checkable outputs with expert-informed rubrics when programmatic verification is insufficient.
- Task taxonomy: The 120 tasks form five top-level families—Office & Productivity, Web & Information, System & Software Operations, Scientific & Engineering, and Multimedia & Design—subdivided into 20 subcategories.The taxonomy uses a two-level structure to show both family proportions and subcategory breakdowns.
4 Benchmark Experiments
Benchmark experiments evaluate multiple terminal-agent frameworks and models using execution-grounded success across repeated trials. Results show strong effects from model capability, agent scaffold, time and reasoning budgets, cost, and task category.
- Experimental setup: Five agent frameworks and twelve models are evaluated, spanning frontier proprietary systems and strong open or widely available alternatives.The frameworks are Terminus-2, Codex, OpenHands, Mini-SWE-Agent, and Claude Code.
- Evaluation protocol: Execution-grounded task success is measured from final environment states, averaging five independent trials per agent–model–thinking configuration.The evaluation also reports Pass@1, Pass@5, and All-5 to characterize repeated-trial reliability.
- Model comparison at a fixed agent framework: 60.1%, 59.7%, and 58.0% are the leading fixed-Terminus-2 results for GPT-5.5, Claude Opus 4.8, and Claude Opus 4.7, respectively.Within Claude, Opus 4.8 reaches 59.7%, Sonnet 4.6 42.8%, and Haiku 4.5 23.9%, indicating tier-sensitive benchmark performance.
- Best achievable performance per agent: 65.8% is the best agent configuration: Claude Code with Claude Opus 4.8 (max), followed by Codex/GPT-5.5 (xhigh) at 64.7%.OpenHands reaches 63.4%, Mini-SWE-Agent 62.4%, and Terminus-2 60.1% with their respective best models.
- Budget and reasoning scaling: 27.1 points is the success-rate improvement from increasing the time limit from 150s to 2400s, while timed-out trials fall from 337 to 4 of 600.For reasoning effort, success rises from 36.5% under none to 60.1% under xhigh; high-to-xhigh adds only 2.3 points while increasing average output from ∼13K to ∼19K tokens.
- Scaffold, cost, and task variation: 61.3% versus 60.2% is the mean success-rate comparison between GPT-5.5 and Claude Opus 4.8 across three open-source scaffolds, but scaffold-dependent reversals make model rankings non-invariant.Across configurations, costs range from roughly $12 to $304 per run and success rates from 23.9% to 65.8%; category averages also conceal substantial task-level heterogeneity.
5 Limitations
TUA-Bench evaluates terminal-based computer use rather than the full spectrum of computer interaction, with coverage constrained by immature CLI or headless support, limited specialized-domain sampling, English-only task descriptions, and risks from public release.
- Coverage limitations: TUA-Bench does not cover the full spectrum of computer interaction because some applications lack mature CLI or headless support.This limits which workflows can be faithfully represented, although growing CLI availability may increase the setting’s relevance.
- Coverage limitations: The professional track samples only a limited set of specialized domains, and all task descriptions are currently English-only.
- Release considerations: Public release may expose task-related limitations.
6 Conclusion
TUA-Bench is introduced as a benchmark for evaluating general-purpose terminal-use agents across diverse everyday and professional workflows. It targets text-native command-line environments through 120 manually curated tasks spanning routine digital work and expert-co-designed domain-specific workflows.
- TUA-Bench evaluates general-purpose terminal-use agents across diverse everyday and professional workflows.
- The benchmark targets text-native command-line environments rather than GUI interaction or programming-centric terminal tasks.
- 120 manually curated tasks span routine digital work and domain-specific workflows co-designed with experts across multiple fields.
- Terminal interaction requires agents to plan, invoke tools, monitor execution, and recover from errors.
Appendix · A Additional Results · A.1 Full Results and Online Leaderboard
Table 5 reports the complete results, ranking all configurations by success rate. A public leaderboard at tuabench.ai is continuously updated with evaluations of newly released models.
- A.1 Full Results and Online Leaderboard: Table 5 contains the complete results for the benchmark.The passage identifies Table 5 as the location of the full result set.
- A.1 Full Results and Online Leaderboard: All configurations in Table 5 are ranked by success rate.The ranking criterion is explicitly stated as success rate.
- A.1 Full Results and Online Leaderboard: The benchmark maintains a public leaderboard at https://tuabench.ai.The leaderboard is publicly accessible at the stated URL.
- A.1 Full Results and Online Leaderboard: The online leaderboard is continuously updated with evaluations of newly released models.Updates incorporate evaluations of models released after the initial results.
- A.1 Full Results and Online Leaderboard: Newly released models can be evaluated and added to the ongoing leaderboard results.The passage describes continuing evaluations rather than a fixed leaderboard.
- A.1 Full Results and Online Leaderboard: The appendix directs readers to both the complete ranked results and the live online leaderboard.Together, Table 5 and tuabench.ai provide static and continuously updated result views.
A.2 Task-Level Results
Task-level analysis reveals persistent failures on some Office layout tasks across configurations, while increasing reasoning effort generally improves rewards but does not do so monotonically. Figure 8 provides the fine-grained agent–model–effort reward matrix, and Figure 9 isolates reasoning-effort effects within agent–model blocks.
- Persistent failure modes and the role of the agent scaffold: Figure 8 presents per-task rewards across all agent–model–effort configurations, exposing persistent failures that appear as near-horizontal red bands.Examples include multiple Office chart- and slide-layout tasks, such as 065-resizeslide3-slide6, 066-set-slide-image-heights, and 067-align-slide-.
- Reasoning effort: Increasing reasoning effort generally shifts task rewards upward within most agent–model blocks, consistent with benefits from additional inference-time computation.Figure 9 fixes each agent–model pair and sweeps effort from none to xhigh, with max available for Claude Opus 4.7.
- Reasoning effort: The effect of reasoning effort is not uniformly monotonic, as many tasks saturate at medium or higher effort levels.The task-level analysis therefore distinguishes broad gains from effort-dependent saturation and irregularities.
B Full Task List
The benchmark’s full task list spans scientific and engineering workflows, multimedia and design activities, and office productivity tasks, each paired with a concrete terminal-agent instruction. Tasks require inspecting supplied inputs, using designated software or defensible workflows, and writing precisely specified artifact files.
- Task inventory: The list enumerates each benchmark task with its category and corresponding instruction prompt.It includes scientific and engineering, multimedia and design, and office and productivity tasks.
- Scientific & Engineering: The nuclei-location task requires a CSV with exactly x,y columns and one pixel-coordinate center per detected nucleus.The workflow identifies the `hoe` file as the nuclear stain, automatically sets smoothing and local-maxima distance, caps detections at 500, and reports CellProfiler-equivalent object centers.
- Multimedia, Design & Productivity: Multimedia, diagram, and office tasks cover video-event counting, frame-range identification, organizational-chart repair, and spreadsheet revenue pivots.Outputs include single-integer or formatted text files, repaired draw.io and PNG files, and an in-place Excel workbook with a Sheet2 Pivot Table.
- Scientific & Engineering: Scientific and engineering tasks include bioimage analysis, energy and thermal simulation, building-model extraction, and medical-image reconstruction.Examples require CellProfiler-based nuclei analysis, OpenStudio/EnergyPlus modeling, OpenFOAM heater placement, and 3D Slicer prostate segmentation.
- Scientific & Engineering: Engineering tasks impose concrete artifact and verification requirements, including annual energy totals checked against hidden ground truth with 1% relative tolerance.Other tasks require numeric JSON for four sensors, an SVG review render, or a reduced OpenStudio model saved under `/app/artifacts`.