PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

Edward Lue Chee Lip, Boden Moraski, Tim Knappe, Lang Xiong, Sarvesh Gharat, Antonio Mari, Ivan Bercovich

arXiv:2609.26826v1cs.LGcs.AI

TL;DR

A zero pass rate can reflect a genuine capability gap or problems in the task and evaluation stack, making all-fail scores ambiguous. The paper audits a frozen agentic-benchmark production record with controls and task evidence; 78 of 125 all-fail tasks survive its validity screens as certified-unsolved candidates. The label is deliberately limited and does not establish intrinsic hardness.

  • Problem

    An all-fail score may reflect a capability gap, underspecification, a broken reference solution, unreliable infrastructure, or a verifier that accepts unintended behavior.

    Source: §1 · p. 1

  • Method

    The authors combine production artifacts, reference and empty-solution controls, adversarial trials, trajectories, telemetry, and review evidence in an ordered validity screen for tasks with no honest pass.

    Source: §1 · p. 1, §2 · p. 3

  • Results

    78 of 125 tasks with no honest pass survive as certified-unsolved candidates, while the remainder require repair, exclusion, or an unresolved label.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Frontier benchmarks should report evidence about reference routes, infrastructure, verifier integrity, harness conditions, and unresolved uncertainty before treating all-fail tasks as capability claims.

    Source: §1 · pp. 1–2

    Oracle coverage is uneven: 53 of 78 certified-unsolved candidates rely on a single recorded reference run, which does not establish verifier completeness or deterministic behavior.

    Source: §6 · p. 7

Abstract

from arXiv · show

Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass rate can come from a real capability gap, but it can also come from missing context, a broken reference solution, infrastructure failure, or a verifier that can be bypassed. In this paper, we study this issue using a frozen Terminal-Bench 3 / Frontier-Bench 0.1 production record with 1,081 pull requests, 639 scored tasks, 28,801 trials, and $105,933 in logged agent spend. We ask what an all-fail task actually certifies. For the 125 tasks with no honest pass, we combine task artifacts, reference-solution runs, empty-solution controls, adversarial trials, trajectories, telemetry, and review records, and apply an ordered validity screen. Only 78 of the 125 tasks survive as certified-unsolved candidates. The remaining tasks include 14 with broken oracles, 8 dominated by infrastructure failures, 4 that are only passable through verifier bypasses, and 21 whose solvability is not certified by the available evidence. Thus, lack of saturation and genuine difficulty are not the same thing. The certified-unsolved label is also narrow: it means that the authored route passed, infrastructure did not dominate, no strict bypass was observed, and all evaluated agents failed. It does not prove intrinsic hardness, verifier completeness, or failure at the intended capability. We further analyze rejected submissions and passing tasks to show that pass rate alone cannot explain why a task is difficult. Overall, our results suggest that frontier benchmarks should report the evidence behind their all-fail tasks before using them as capability claims.

1 Introduction

An all-fail score can reflect a capability gap or defects in the task and evaluation stack, so zero pass rate alone does not establish genuine difficulty. The paper audits production evidence to distinguish these cases and characterize difficulty beyond pass rate.

1 Introduction

  • Agentic tasks combine instructions, files, dependencies, tools, environments, reference solutions, and verifiers, so failures may arise outside the intended capability.Pass rate alone cannot identify whether the agent failed at the intended capability or the task failed as a measurement instrument.Source: §1 · p. 1
  • The frozen production record includes 1,081 pull requests, 639 scored tasks, 28,801 trials, and artifacts such as controls, trajectories, and rejected submissions.The corpus is used to ask what evidence supports each task-level failure.Source: §1 · pp. 1–2
  • 78 of 125 tasks with no honest pass survive the validity screen as certified-unsolved candidates; 47 require repair, exclusion, or an unresolved label.The other outcomes are 14 broken-oracle tasks, 8 infrastructure-limited tasks, 4 exploit-only-passable tasks, and 21 with uncertified solvability.Source: §1 · p. 1
  • Among tasks with an honest pass, minimum output tokens to first archived pass separate broader genuine-hard candidates from tractable tasks with AUC 0.750.The authors describe this as an observational effort signal, not a definition of intrinsic hardness.Source: §1 · p. 2
  • The authors analyze 555 closed-unmerged submissions, finding many failed candidate tasks were rejected for issues including ambiguity, verifier problems, and nondeterminism.They argue frontier benchmarks should report evidence behind all-fail tasks before treating them as capability claims.Source: §1 · pp. 1–2
  • The authors contrast adjudicated production-evidence labels with related item-response approaches that infer difficulty and feasibility from the same responses used to score items.They also treat scores as conditional on the recorded task, harness, verifier, and environment.Source: §1 · p. 2

2 Corpus and Adjudication Protocol

The protocol treats the production archive as a task-production corpus and joins task artifacts, trials, controls, telemetry, trajectories, and verifier-integrity evidence. It applies ordered checks to all-fail tasks because pass rate alone cannot establish feasibility or explain failure.

2 Corpus and Adjudication Protocol

  • The study uses a frozen production record of 1,081 pull requests, 639 scored tasks, 28,801 trials, and $105,933 in logged agent spend.The archive links task packages and review histories with verifier and reference-solution runs, telemetry, trajectories, and controls.Source: §2 · p. 2
  • The audit joins ordinary trials with reference-solution, empty-solution, and cheat-oriented controls, then combines them with telemetry, trajectory, and verifier-integrity evidence.No single signal is treated as ground truth; the joined evidence passes through an ordered screen.Source: Figure 1 · p. 3, §2 · p. 3
  • Observed hardness is separated into difficulty, measured descriptively by realized output-token effort, and feasibility, whether a correct solution is reachable and accepted as shipped.A zero pass rate does not establish feasibility, so the available task evidence and controls must be adjudicated.Source: §2 · p. 3
  • An oracle tests whether the authored reference route is accepted, a nop tests rejection of an empty solution, and a cheat-variant probes verifier bypasses.A passing oracle establishes one accepted route but not verifier completeness; a failing nop rules out only the simplest empty-solution leak.Source: §2 · p. 3
  • For all-fail tasks, the first matching rule checks reference-route failure, infrastructure dominance, strict bypass evidence, and repeated oracle success before assigning an unresolved label.A certified-unsolved candidate has a passing authored route, no infrastructure-dominance or strict-bypass disqualifier, and all evaluated ordinary agents failing in the frozen setup; this does not prove intrinsic hardness or verifier completeness.Source: §2 · p. 3
  • The ordered validity screen labels 125 tasks with no honest pass: 78 certified-unsolved candidates, 14 broken oracles, 8 infrastructure-limited, 4 exploit-only-passable, and 21 with uncertified solvability.Figure 2 presents the five outcomes as a partition of the all-fail tasks.Source: §2 · p. 3, Figure 2 · p. 4

3 What an All-Fail Score Can Support

Among 125 tasks with no honest pass, adjudication separates 78 certified-unsolved candidates from 47 tasks that require repair, exclusion, or unresolved status. Certification is stronger than a raw zero but does not establish intrinsic hardness; its interpretation depends on the evidence and checks available for each task.

3 What an All-Fail Score Can Support

  • 78 of 125 all-fail tasks are certified-unsolved candidates; the other 47 comprise 14 broken-oracle, 8 infrastructure-limited, 4 exploit-only-passable, and 21 uncertified-solvability tasks.The partition supports different maintainer actions: retain certified candidates, repair or exclude defective cases, and leave uncertified cases unresolved.Source: §3 · p. 4
  • Certification rises from 85 tasks with an observed oracle pass to 79 with clean infrastructure and 78 with no strict bypass; requiring at least two oracle runs leaves 25.Table 1 presents these as successive requirements, with the final row a stronger stability cut rather than independent-implementation evidence.Source: §3 · p. 4, Table 1 · p. 4
  • 53 of the 78 certified-unsolved candidates have only one recorded reference run, which establishes one accepted route once but not verifier completeness or failure at the intended crux.Repeated reference runs and independent implementations would strengthen the claim.Source: §3 · p. 4
  • Nearby infrastructure thresholds change only a small number of task IDs, while the headline pattern remains a large certified group and a substantial group needing repair, exclusion, or unresolved status.The threshold is not unique, and the adjudication checks are evidence-qualified rather than a single accuracy estimate.Source: §3 · p. 4, Table 2 · p. 5
  • The labels are auditable decisions supported by controls, trajectories, telemetry, and review evidence, not claims that certified tasks are impossible.A benchmark zero should be interpreted only after the evidence behind it is visible.Source: §3 · pp. 4–5
  • The validity screen and depth question are distinct: agents may spend most of their time on setup or dependency resolution even when the authored route passes.Cases remain unresolved when trajectories do not identify the failure layer.Source: §3 · p. 5

4 How Misleading Difficulty Enters Task Production

Rejected submissions show that agent struggle does not by itself make a task valid frontier evidence: apparent difficulty can come from ambiguity, verifier defects, nondeterminism, or other task-quality problems. Production history and review records help identify those sources and distinguish them from genuine difficulty.

4 How Misleading Difficulty Enters Task Production

  • Among 555 closed-unmerged submissions, insufficient difficulty is the largest rejection category, but many others reflect ambiguity, verifier problems, nondeterminism, or quality and metadata issues.Figure 3 labels primary rejection reasons from review threads using a single-pass LLM judge.Source: §4 · p. 5, Figure 3 · p. 6
  • Ambiguous or underspecified tasks average a 0.17 honest pass rate and broken or nondeterministic tasks 0.18, near the 0.10 for verifier-overfit or too-hard tasks.Pass rate contains difficulty information but cannot distinguish these causes on its own.Source: §4 · p. 5
  • Hackable-verifier rejections average a 0.41 pass rate and quality or metadata rejections 0.45, so invalid benchmark items need not have low pass rates.Pass rate can miss defects that do not strongly affect solve rate.Source: §4 · p. 5
  • Among 349 rejected pull requests that reached the trial harness, about 42% have pass rate at most 0.2; after excluding clearly defective verifier evidence, the failed-by-agents share rises to about 45%.Many rejected tasks challenge agents, but that struggle does not cleanly support a frontier capability claim.Source: §4 · p. 5
  • Pull-request revisions, reviewer objections, control outcomes, and trial failures leave different traces for defects such as missing files, ambiguous instructions, route-coupled verifiers, and fragile environments.Preserving these traces helps task authors see which checks caught a defect and why.Source: §4 · p. 5
  • The rejection record complements the all-fail screen by showing how many apparent zeros must be filtered before they become benchmark evidence.The screen evaluates zeros after acceptance; rejection history captures filtering earlier in task production.Source: §4 · p. 6

5 Cost Beyond Pass Rate

The paper tests whether the cheapest recorded success provides a graded effort signal beyond pass rate, which cannot measure successful-solution cost for all-fail tasks. It finds higher minimum successful token cost among broader genuine-hard candidates, while treating that signal as observational.

5 Cost Beyond Pass Rate

  • For 346 tasks with at least one honest pass, the analysis compared 217 broader genuine-hard candidates with 129 tractable tasks using minimum output tokens from any archived honest passing trial.This is the cheapest observed success, not the average cost across attempts.Source: §5 · p. 6
  • 24,829 versus 12,388 median minimum successful output tokens separated broader genuine-hard candidates from tractable tasks; AUC was 0.750 (95% CI [0.700, 0.801]).The comparison covers tasks with at least one honest pass and excludes verifier-suspect tasks.Source: §5 · p. 6
  • Minimum tokens to first pass is observational, mixes reasoning and recording conditions, and exists only for tasks where a pass was observed.The measure also reflects verbosity, tool use, harness policy, model assignment, retry behavior, budget, and task revision.Source: §5 · p. 6
  • A controlled hint ladder is proposed as a follow-up, not reported here; it would hold task and evaluation conditions fixed while adding preregistered information across repeated trials.The experiment would require owner approval, compute, and domain review.Source: §5 · p. 6
  • Token cost complements the validity audit by distinguishing whether failure can be interpreted as evidence from how much recorded effort success required.Together, the audit and cost signal provide a view beyond pass rate alone.Source: §5 · p. 6

6 Discussion and Limitations

The discussion argues that all-fail scores need inspectable evidence and that benchmark comparisons remain conditional on the instrument and frozen setup. It also identifies limits in the validity labels, trajectory evidence, observational run data, and rejection taxonomy.

6 Discussion and Limitations

  • Reporting reference-route success, empty-solution controls, infrastructure status, verifier-integrity checks, and whether failures reached the claimed crux makes benchmark failures more inspectable.These checks can expose task defects before they are mistaken for capability gaps.Source: §6 · p. 7
  • Model comparisons remain conditional on task, environment, verifier, model, harness, provider state, budget, and retry policy; the paper does not claim context-free rankings.Pinned instruments and reported unresolved cases make comparisons more useful.Source: §6 · p. 7
  • The audit addresses whether a shipped item supports its capability claim, not which capabilities benchmark builders should prioritize.Publishers should state why progress on their chosen construct matters.Source: §6 · p. 7
  • Oracle coverage is uneven, and 53 of 78 certified-unsolved candidates rely on a single recorded reference run, which does not establish verifier completeness or deterministic behavior.Repeated reference runs and independently implemented solutions would strengthen certification.Source: §6 · p. 7
  • When the archive cannot identify whether failure arose at the claimed crux or from setup, dependencies, or incomplete records, the case remains unresolved rather than being treated as hardness.An indeterminate case is not halfway between genuine-hard and fake-hard; evidence is insufficient.Source: §6 · p. 7
  • Model and harness evidence is observational rather than fully crossed, and token cost mixes reasoning, verbosity, tool use, retry behavior, and harness policy.The AUC is therefore a descriptive signal, not a causal estimate of intrinsic difficulty.Source: §6 · p. 7
  • Each rejected pull request receives one primary reason despite potentially co-occurring defects, and review labels reflect records and a single-pass LLM-judge protocol rather than duplicate expert annotation.The labels describe recorded production decisions, not unique causal mechanisms; provenance metadata is incomplete.Source: §6 · p. 7
  • All findings concern one frozen snapshot of an evolving benchmark, while the controlled hint ladder and matched-harness experiments remain follow-up work.Later repairs, tasks, model runs, or harness changes can alter raw counts.Source: §6 · p. 7

7 Conclusion

The conclusion holds that all-fail scores support different conclusions depending on what produced the failure. It presents the audit as a way to make frontier-benchmark failures more inspectable without treating certified-unsolved status as proof of intrinsic hardness.

7 Conclusion

  • Of 125 all-fail tasks in the frozen record, 78 survived as certified-unsolved candidates, while the rest required repair, exclusion, or more evidence.The certified label records a working reference route and no visible infrastructure or strict-bypass disqualifiers, but does not prove intrinsic hardness.Source: §7 · p. 7

Generative AI Disclosure

The disclosure separates language editing and formatting from preliminary reviewer feedback, while assigning technical work and conclusions to the authors.

Generative AI Disclosure

Related papers

Loading 2609.26826v1…