Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

• The benchmark retains 46 original tasks, fixes task and harness defects, and adds 31 tasks, yielding 77 tasks across 13 domains and eight task types.The evaluation infrastructure isolates checkpoint grading, clears verifier artifacts, audits successful trajectories, and retains clean attempts.Source: §1 · p. 2
• A reconstruction pipeline turns parameterized scientific or engineering artifacts into tasks using public input–output evidence and withheld scenarios for grading.The agent receives tools and evidence but not the reference implementation.Source: §1 · p. 2
• Across 1,145 scored model–task cells, GPT-6-astra leads with dense mean reward 0.69 and 37 tasks at R ≥0.95, followed by GPT-5.6-sol at 0.67 and 32 tasks.The comparison covers 15 frontier models.Source: §1 · p. 2
• On 30 reconstruction tasks, binary verifier scores average 0.221, compared with 0.653 under dense ladder scoring.The paper reports that outcome-only evaluation misses substantial intermediate progress.Source: §1 · p. 2
• Three-hour runs with verifier checkpoints every 30 minutes enable analysis of sustained progress, late completion, and regression within a trajectory.The run budget increases from 90 minutes.Source: §1 · p. 2
• The evaluation protocol fixes task and harness bugs, isolates verifier execution, audits grader access, and replaces affected results with clean reruns.Source: §1 · p. 2

2.1 Task formulation

• Each task combines a natural-language instruction, containerized terminal environment, task and runtime configuration, and deterministic verifier or simulator; the instruction is the agent’s only specification.Agents can inspect files, edit programs, run commands, and evaluate intermediate outputs until success or the time budget.Source: §2.1 · pp. 2–3

2.2 Benchmark extension and splits

• The 77 tasks preserve all 46 original identities and descriptions while adding 31 tasks, split between released workflows and reconstruction tasks.Released workflows expose dense rewards; reconstruction tasks infer a parameterized artifact’s behavior from public observations and are graded on hidden scenarios.Source: §2.2 · p. 3

2.3 Evaluation hardening

• Three-hour runs with verification every 30 minutes can reveal regressions and expose defects such as stale verifier artifacts or hidden grader data leaking across phases.The protocol fixes harness defects, isolates verifier phases, clears logs, and audits successful trajectories for hidden-data access.Source: §2.3 · p. 3

2.4 Domains

• The benchmark spans 13 domains, including the original areas and newer domains such as geospatial workflows, life sciences, and physical simulation.Figure 1 summarizes the task distribution across these domains.Source: §2.4 · p. 3, Figure 1 · p. 4

3.1 Construction examples

• Public evidence includes artifacts such as LiDAR point clouds, molecular diagrams, and acoustic waveforms, which expose changes in the reference artifact’s behavior.The examples show density and amplitude changes, reaction and descriptor behavior, and simulated impulse-response changes.Source: §3.1 · p. 4
• A gated ladder awards partial progress within the furthest rung reached, with each rung requiring completion of the one below.Source: §3.1 · p. 5
• Ten of 15 models reached the private oracle yet scored 0 under binary grading; the example reports five passes and ten runs flattened to zero.Source: §3.1 · p. 5
• Reconstruction tasks sweep a working reference artifact over meaningful parameters, expose behavioral examples, and withhold scenarios for grading the agent’s implementation on unseen inputs.The figure applies the same task contract to point clouds, molecules, and waveforms.Source: Figure 2 · p. 5

3.2 Task types

• LHTB 2.0 includes eight task types; reconstruction is the new automated construction family, while the remaining types preserve the original benchmark’s heterogeneous workflows.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Quality control

• Task inclusion requires a reproducible reference solution scoring R = 1.0 on hidden cases, nontrivial starter materials, and instruction–verifier alignment review.DeepSeek-V4-Pro reviews whether instructions enable satisfying the grader without revealing hidden answers.Source: §3.3 · p. 6

4.1 Main Results

• GPT-6-astra softly solves 48.1% (37/77) of tasks, ahead of GPT-5.6-sol at 41.6% (32/77), while many models earn substantial partial reward without passing.GPT-6-astra leads on reconstruction tasks (mean reward 0.93; 24 solves), while GPT-5.6-sol slightly leads on released workflows (0.57 vs. 0.54).Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Success and Failures

• Systems, performance & security has the highest category mean (0.73), while scientific computing & simulation (0.30) and multimodal & imaging analysis (0.31) score lowest.No model solves 24 tasks, 20 from the released split; 12 of those are audit-and-repair tasks, and eight of the nine tasks where no model exceeds R = 0.2 are audit-and-repair.Source: §4.2 · p. 8

4.3 Solve Rate and Reward over Time and Steps

• 42% of solves arrive after one hour and 21% after two; a one-hour limit would rank Hunyuan hy4 ninth rather than fourth and DeepSeek V4 Pro 13th rather than eighth.The median successful run solves after 57 minutes; GPT-6-astra is faster, with a median solve time of 24 minutes.Source: §4.3 · pp. 8–9, Figure 5 · p. 9
• Step latency ranges from 23 seconds for MiniMax M3 to 141 seconds for Hunyuan hy4, and GPT-6-astra solves 22 tasks within 50 steps versus 14 for GPT-5.6-sol.Models spend 48–76% of each run waiting for the model; more steps do not guarantee more reward.Source: §4.3 · p. 9, Figure 5 · p. 9, Figure 6 · p. 10
• 92% of completion claims occur on runs that never pass verification, with GPT-6-astra and GPT-5.6-sol making first claims after median times of 7 and 8 minutes.On unsolved runs, MiniMax M3 averages 58 claims, compared with 24 for GPT-6-astra and 17 for GPT-5.6-sol.Source: §4.3 · pp. 9–10, Figure 6 · p. 10

4.4 Time and Process Reward Analysis

• Mean reward rises from 0.29 at 30 minutes to 0.43 by run end, while 65 trajectories regress by more than 0.05 from a checkpoint.After removing successful runs, every model’s mean reward at 180 minutes remains above its 60-minute level.Source: §4.4 · p. 10, Figure 7 · p. 11

Per-task costs

• Models average 262.5 LLM calls and 94.3 minutes per task, at an estimated $48.4 per task.Costs use per-task token usage and public list prices; tokens per task include input and output.Source: Table 2 · p. 11

Cost–reward frontier

• GPT-6-astra achieves the highest pass rate at the highest total cost, while Hunyuan hy4 anchors the low-cost frontier and GPT-5.6-sol offers a cheaper, strong second tier.At the soft R ≥0.95 criterion, Kimi K3 also joins the frontier.Source: §5 · p. 12

Cost estimates

• The estimated full-suite cost ranges from about $285 for Hunyuan hy4 to about $30,581 for GPT-6-astra.The estimates are reconstructed from per-task token usage and public list prices.Source: §5 · p. 12

Spending by outcome and hour

• Between 83% and 99% of each model’s spend goes to tasks it never solves; benchmark-wide spending rises from 18% in hour one to 51% in hour three, while solves are front-loaded.GPT-6-astra averages $664 per unsolved task versus $101 per solved task; a third-hour solve costs about eight times a first-hour solve ($708 versus $90).Source: §5 · p. 12

Analysis approach

• The analysis parses shell commands from 1,135 full-trajectory runs and compares how models work through tasks by their command-type mix.The Figure 11 comparison reports that higher-reward runs tend to edit, test, and inspect more, while repeated-command loops correlate with lower reward.Source: §6 · p. 12, Figure 11 · p. 14

Command and tool overlap

• Runs from different models on the same task share only 3% of exact commands and 18% of tool signatures on average, despite a 0.63 cosine similarity in overall program mix.The task largely determines tool choice, while the specific command sequence depends strongly on the model.Source: §6 · pp. 12–13

Command patterns and reward

• More file-editing commands tend to accompany higher reward, while long repeated-command streaks tend to accompany lower reward.Among 800 runs that exhaust the three-hour budget without solving, these statistics do not reliably distinguish greater from lesser progress.Source: §6 · p. 13, Figure 11 · p. 14

Correlation analysis

• Overall program diversity matters little in the reported command-statistic correlations with reward across models on the same task.The two Gemini submissions are excluded because they ran under a different harness.Source: §6 · p. 14

Long-horizon terminal tasks and trajectories

• Long-horizon benchmarks raise task difficulty and horizon, but validating realistic tasks, reference solutions, and trajectories is increasingly expensive and time-consuming [5] [18], [11], [3].Performance also depends on the tool-providing, context-managing, and feedback-handling harness [9] [29].Source: §7 · p. 14

Long-horizon task evaluation and training

• Dense, verifiable process rewards credit intermediate achievements, while repeated checkpoints reveal whether progress continues, stalls, or is lost [12] [15].The cited work connects intermediate rewards to sparse credit assignment in reinforcement learning [27] [28].Source: §7 · p. 14

8 Conclusion

• LHTB 2.0 extends the original benchmark to 77 terminal tasks spanning 13 domains and evaluates 15 frontier models with a shared terminus-2 agent.Outcome-only evaluation misses substantial progress: process reward can rise steadily on runs that never fully pass, and much cost accrues in the final hour of unsolved runs.Source: §8 · pp. 14–15
• Only 192 of 1,145 scored runs (16.8%) pass overall, and GPT-6-astra resolves 48.1% of tasks (37/77) at R ≥0.95.Many runs continue improving over long horizons, including runs that never pass.Source: §8 · p. 15

A Task Inventory

• The 77-task inventory preserves the original 46 tasks and adds 30 reconstruction tasks plus one optimization task across the paper’s 13-domain taxonomy.The first 46 rows preserve the original suite; the remaining 31 are LHTB 2.0 additions.Source: A Task Inventory · p. 17
• Tasks are labeled Easy at dense mean reward ≥0.5 and Hard below that threshold, based on model results in Figure 3.Source: A Task Inventory · p. 17

Scientific computing & simulation

• Scientific and analytical tasks include finite-element plate stress, stochastic convergence testing, hydrograph modeling, multi-well log correlation, and X-ray diffraction peak-shift analysis.Source: A Task Inventory · p. 20

Document, OCR & data engineering

• Document and data tasks cover HarfBuzz shaping, Ghostscript PDF/A repair, SQLite WAL recovery, and Tesseract table OCR; GDAL tasks reconstruct antimeridian clipping and VRT mosaics.Source: A Task Inventory · pp. 20–21

Geospatial and imaging analysis

• Geospatial and imaging tasks reconstruct LiDAR registration, map-label placement, road generalization, and medical slice-series geometry.Source: A Task Inventory · p. 20

Systems, performance & security

• Systems and security tasks reconstruct nginx routing, OPA/Rego authorization boundaries, OpenSSL name-constraint validation, and HTTP/2/TLS fingerprint extraction.Source: A Task Inventory · p. 20

Life sciences, chemistry & materials

• Life-sciences tasks include PDB structural RMSD, mass-spectrometry retention-time alignment, and samtools mpileup consensus calling.Source: A Task Inventory · pp. 20–21

Physical simulation, signals & control

• Physical simulation and control tasks reconstruct robotic pick-and-reach trajectories and seismic waveform cross-correlation lags.Source: A Task Inventory · p. 21