Source-linked AI summary
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, Leowei Liang
TL;DR
Existing terminal benchmarks provide limited evidence about agents’ performance on long workflows because they focus on short tasks and final outcomes. This paper introduces Long-Horizon-Terminal-Bench, which uses graded subtasks and dense partial rewards across diverse terminal tasks; agents still require substantial effort and achieve low resolution rates, exposing room for improvement in long-horizon execution.
Problem
Existing terminal benchmarks focus on short tasks and final-state outcomes, providing sparse evidence about intermediate progress in long-horizon workflows.
Method
Long-Horizon-Terminal-Bench evaluates 46 containerized terminal tasks across diverse domains using graded subtasks, intermediate checks, and partial-credit rewards.
Results
Across 17 frontier models, tasks averaged 239 episodes, 9.8M tokens, $10.8 in API cost, and 88.9 minutes per run, while Grok 4.5 resolved 28.3% at R ≥0.95 and the mean pass rate was 6.4%.
Takeaways & Limitations
Dense rewards reveal meaningful but incomplete progress and distinguish timeout-driven incompletion, premature stopping, and weak self-verification.
Abstract
from arXiv · showhide
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
1 Introduction
Existing terminal benchmarks emphasize short tasks and binary final outcomes, leaving long workflows and partial progress poorly measured. Long-Horizon-Terminal-Bench addresses this with 46 long-horizon terminal tasks, dense subtask rewards, and analyses of sustained execution failures.
- Motivation: Real workflows require hundreds of steps, evolving plans, long-context management, iterative debugging, and recovery from mistakes.Examples include research reproduction, repository setup, multimodal dataset auditing, compiler debugging, and multi-stage ML pipelines.
- Benchmark gap: Existing benchmarks use short horizons and final-state grading, so agents making substantial intermediate progress can receive the same score as agents failing immediately.These benchmarks therefore provide sparse evidence about long-horizon navigation, step revision, and multi-stage debugging.
- Benchmark design: Long-Horizon-Terminal-Bench contains 46 containerized tasks across nine categories, decomposes workflows into graded subtasks, and awards credit for partial progress.Intermediate checks provide dense feedback while preserving terminal-task realism.
- Evaluation scale: 239 episodes, 9.8M tokens, and 88.9 minutes are average rollout demands across 17 frontier agents under a 90-minute timeout.The strongest reported configuration reaches 28.3% success at a 0.95 reward threshold, while the mean pass rate is 6.4%.
- Failure analysis: Failure analysis indicates that agents struggle to sustain progress, verify completion, and finish within budget rather than failing at every local step.Dense rewards distinguish timeout-driven incomplete progress from premature stopping and weak self-verification.
2 Long-Horizon-Terminal-Bench
Long-Horizon-Terminal-Bench extends containerized terminal evaluation with interactive, multi-step workflows and subtask-based grading. Its tasks cover diverse scientific, engineering, software, and multimodal settings while using hidden stress cases to reward robust end-to-end solutions.
- Task formulation: Each task provides a natural-language goal, Docker environment, configuration, and oracle or simulator, with agents operating entirely through terminal commands over hundreds of steps.Agents inspect and modify files, run scripts, and examine intermediate outputs until success or timeout.
- Task formulation: Long-horizon structure comes from multi-stage domains and intermediate targets that require dozens to hundreds of distinct operations over minutes to hours.The benchmark explicitly factors workflows into intermediate targets discovered and executed over time.
- Subtask-based grading: The deterministic grader scores weighted subtasks with normalized rewards, using objective evidence from final files, outputs, tests, or simulator responses.Subtasks may be binary, continuous or thresholded, or aggregated across episodes.
- Subtask-based grading: Mean task reward complements thresholded resolution by measuring how far agents progress on tasks they do not fully solve.Resolution is determined by whether overall reward reaches the selected threshold.
- Robustness: Hidden stress cases vary schemas, missing values, noise, image transformations, and coordinate conventions, requiring robust implementations beyond public examples.Public checks have relatively low reward weight, while hidden cases test generalization and end-to-end artifact generation.
- Task coverage: The benchmark spans 46 tasks across 21 high-level categories, with interactive games and multimodal audits as the largest groups and many scientific and engineering workflows represented.The category distribution is broad rather than dominated by one domain.
3 Experiments
Experiments show that frontier agents remain weak on long-horizon terminal workflows despite dense partial-credit grading. Performance, cost, and failure analyses indicate that completion within the time budget—not merely local execution correctness—is the central bottleneck.
- 3.1 Main Results: Grok 4.5 leads the 46-task benchmark at 28.3% pass@1 for R ≥0.95, while GPT-5.6-sol and GPT-5.5 each reach 15.2%.Grok 4.5 also leads mean normalized reward at R = 0.51.
- 3.2 Dense Reward Is Necessary to Rank Models: Under R ≥1.0, 10 of 17 models solve zero tasks, while dense grading records substantial partial progress among many otherwise failing runs.Only 6.4% of runs pass R ≥0.95, but 62.8% achieve partial reward.
- 3.2 Dense Reward Is Necessary to Rank Models: Near-misses occur nearly twice as often as passes, with 90 runs at 0.75 ≤R < 0.95 versus 50 passes.Kimi K2.6 solves zero tasks at R ≥0.95 but records five near-misses, including R = 0.94 on grammar-fuzz-coverage-hunt.
- 3.3 Cost Analysis: Grok 4.5 achieves the highest pass rate at about $11 per task, while GPT-5.4 costs about $26 yet has a much lower pass rate.GPT-5.4 requires 302 episodes per task versus 208 for GPT-5.5 at comparable per-million-token pricing.
- 3.4 Dense Rewards Expose Different Failure Patterns: Timeouts account for 79% of unresolved runs, whereas early exits account for 19% and harness errors for 3%.Timed-out runs have mean rewards of only 0.10 to 0.35, while early exits expose weak self-verification and false finishes.
- 3.4 Dense Rewards Expose Different Failure Patterns: The main bottleneck is long-horizon completion: agents can perform many locally correct actions but fail to finish artifacts before the horizon expires.Reducing redundant exploration, preserving state, and avoiding repeated verification loops may improve reward accumulated within a fixed budget.
4 Related Work
Prior work advances interactive terminal and software-engineering agents, measures autonomy through task duration, and explores finer-grained evaluation signals, but long-horizon partial progress remains insufficiently characterized.
- Recent benchmarks evaluate agents on interactive, execution-grounded tasks in realistic work environments rather than only static questions or isolated programming exercises.
- Long-horizon autonomy research treats the duration of tasks agents can complete reliably as a first-class capability variable.
- Existing analyses report that extended trajectories can degrade through accumulated execution errors despite strong short-horizon competence.
- Process-reward and rubric-based methods provide intermediate or multidimensional signals, but much of this work uses fine-grained evaluation primarily as training supervision.
- Benchmark outcomes depend on both the underlying model and the agent harness mediating interaction with the environment.
5 Conclusion
The paper introduces a 46-task benchmark spanning nine domains that uses deterministic subtasks and dense partial-credit signals to evaluate long-horizon progress. Across 17 models, tasks demand substantial execution effort, remain far from saturated, and expose completion and verification bottlenecks.
- 46 containerized terminal tasks span nine domains and are designed to stress long-horizon execution.
- Deterministic, environment-grounded subtasks provide dense partial-credit signals that measure both task completion and progress toward completion.
- 239 episodes, 9.8M tokens, $10.8 in API cost, and 88.9 minutes per run quantify the substantial effort required across 17 frontier models.
- A 28.3% pass rate at R ≥0.95 for the strongest model, with a 6.4% mean across models, indicates that the benchmark remains far from saturated.
- Analysis identifies reliable long-horizon completion as a bottleneck involving timeouts, weak self-verification, planning, memory, progress tracking, and stopping decisions.
A List of Tasks in Long-Horizon-Terminal-Bench
The benchmark task list organizes 46 tasks by identifiers, categories, descriptions, and coarse difficulty labels derived from average task reward. Tasks cover software, scientific, engineering, multimodal, game, and puzzle workflows with varied long-horizon objectives.
- Tasks are labeled Easy when mean reward is at least 0.5 and Hard otherwise.
- Table 2 lists each task’s ID, paper-level category, one-sentence description, and coarse difficulty label.
- Research reproduction & ML: The task set includes research reproduction and machine-learning workflows such as Foldseek-style structural protein search and UNISON fat-tree simulation reproduction.
- Other tasks require iterative software repair, data-quality and imaging audits, adversarial or security-oriented construction, and turn-by-turn game or constraint-puzzle solving.
- Scientific computing & simulation: Scientific-computing and earth, climate, and energy tasks include power-flow checks, N-body acceleration, groundwater simulation audits, raster regression, and renewable-dispatch auditing.