PaperlayerView exact revision on arXiv

Source-linked AI summary

Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation

Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, Yuangang Li, Shanda Li, Zizhao Chen, Hanwen Xing, Harsh Raj, Bo Chen, Quan Shi, Steven Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang, Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyu Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, Przemysław Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Boqin Yuan, Michael Glass, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun, Mao, Yunfei Zhao, Yuxin Wang, Yuxuan Tang, Zhenheng Tang, Zhifei Li, Ziruo Wang, Ziyu She, Kaiyuan Liu, Iheb Chaabane, Yuxin Tang, Xiangyi Li, Satya Sai Srinath Namburi GNVV, Xinyue Zheng, Andy Konwinski, Boxuan Li, Leon Liangyu Chen, Alex Dimakis, Nicholas Carlini, Soroush Vosoughi, Sanmi Koyejo, Di He, Etash Guha, Benjamin Feuer, Mike Merrill, Ludwig Schmidt, Alex Shaw

arXiv:2609.04298v3cs.AIcs.CL

TL;DR

Agentic benchmarks are heterogeneous and difficult to integrate, making broad, comparable evaluation challenging. Harbor Adapters provide a shared infrastructure for benchmarks and agents, which the paper uses to evaluate models and curate Harbor-Index. Across the evaluation, base-model capability is the primary driver of performance gaps, while the curated index contains 82 tasks across 29 benchmarks.

  • Problem

    Agentic benchmarks vary in task formats, environments, interaction protocols, and scoring, making integrations costly and comparisons across domains difficult.

    Source: §1 · p. 2

  • Method

    Harbor Adapters standardize benchmark tasks under Harbor and connect them to agents, supporting a large evaluation and a curated Harbor-Index set.

    Source: §1 · p. 2

  • Results

    Across the evaluation, base-model capability is the primary driver of performance gaps; benchmark rankings often correlate, stronger models use fewer tokens, and unresolved tasks often reflect task-design flaws.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    Harbor-Index 1.0 distills the suite into 82 compact, difficult, diverse, audited tasks spanning 29 benchmarks.

    Source: §6 · p. 9

    The evaluation covers only 54 collected adapter benchmarks and a focused set of popular models and harnesses, so its findings are not universal across LLMs, agent architectures, or settings.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.

1 Introduction

Agentic benchmarks are heterogeneous and costly to integrate, limiting reliable cross-domain comparisons. Harbor Adapters standardize benchmark tasks and connect them to agents; the paper evaluates models and harnesses at scale and introduces a curated task set.

1 Introduction

  • Harbor Adapters port more than 80 benchmarks, including non-agentic datasets made agentic through tools and executable environments, and validate adaptations with code review and parity experiments.The shared Harbor interface changes integration effort from O(mn) benchmark–agent pairings to O(m + n) adapters and agent integrations.Source: §1 · p. 2
  • Across 54 benchmarks, model capability is more influential than harness choice; model rankings are usually correlated, stronger models use fewer tokens, and unresolved tasks often contain specification errors.The evaluation covers 8 models, 16 model–harness configurations, and 6,627 tasks.Source: §1 · pp. 2–3
  • Harbor-Index distills 82 difficult, diverse, high-quality tasks from 29 benchmarks; no evaluated configuration exceeds 30% pass rate, and GPT-5.5 with Codex reaches 28.0%.The set is designed to be affordable while retaining challenge and breadth.Source: §1 · p. 3

2 Harbor Adapters: Infrastructure

Harbor Adapters translate benchmark-specific formats into Harbor’s shared task schema, separating agent execution from benchmark logic. The infrastructure supports audited adaptations and a large evaluation across models, harnesses, benchmarks, and tasks.

2 Harbor Adapters: Infrastructure

  • Harbor Adapters translate dataset-specific formats into Harbor’s instruction, environment, tests, and solution schema, reducing integration from O(mn) to O(m+n).The infrastructure also supports GPU interaction, LLM-as-a-judge verifiers, custom metric aggregation, and multiple sandbox backends.Source: §2 · p. 3
  • Adapters undergo multi-trial parity experiments and a three-stage bot, junior, and senior quality audit to check adaptation fairness and fidelity.The audit process generated over 10,000 GitHub comments across adapters.Source: §2 · p. 3
  • The large-scale evaluation covers 6,627 tasks from 54 benchmarks, with 8 models, 16 model–harness configurations, 4 harness implementations, and 3 trials per setting.The runs yield approximately 0.3 million trajectories.Source: §2 · p. 3

3 Analysis: Agentic Benchmarking at Scale

The analysis compares benchmark progress and redundancy, model and harness effects, performance and cost, and agent failure modes. It finds that model capability outweighs harness design, while task difficulty and execution costs shape evaluation outcomes.

3.1 Benchmark: progress over time and redundancy analysis

  • Of 54 benchmarks, 13 are largely saturated above 90%, while software engineering ranges from near-saturated coding to still-hard speedup tasks.Figure 2 compares current best scores with publication-time state of the art across 39 benchmarks.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • Benchmark scores vary along few shared directions, and difficulty does not determine uniqueness: informative benchmarks span the difficulty spectrum, while some hard ones are predictable from others.After 12 benchmarks are selected, every remaining benchmark correlates at ρ ≥0.7 with a selected one; specialized capabilities can resist prediction.Source: §3.1 · p. 4
  • Across 52 benchmarks, three representative tasks recover system rankings at mean ρ ≈0.923, but recovery is weaker when task types are diverse.The reported examples are WideSearch (ρ = 0.99) and CyberGym (ρ = 0.75).Source: §3.1 · pp. 4–5

3.2 Harness: the interaction analysis between harness and base model

  • The model fixed-effect range is 0.451, 5.2× the harness range of 0.087, although harness design still affects performance.Benchmark difficulty accounts for most score variance (ICC = 0.75).Source: §3.2 · p. 5

3.3 Efficiency: the tradeoff analysis between performance and cost

  • Frontier-model performance gains peak on medium-difficulty tasks, but weaker models cost 2–3× less per trial and token savings do not erase the frontier cost premium.The absolute cost gap peaks at $0.37 per trial in the 0.7–0.8 difficulty bucket.Source: §3.3 · pp. 5–6
  • Frontier models use fewer tokens at every difficulty tier; on easiest tasks, they use 42% of weaker-model tokens, whose extra usage often reflects misunderstanding and redundant tool calls.Manual inspection reports 6.14× more tokens on SimpleQA and 2.28× on KUMO for weaker models.Source: §3.3 · p. 5

3.4 Bottlenecks: failure mode analysis

  • Wrong factual answers, algorithmic bugs, and hidden-test regressions dominate frontier-model failures; native harnesses support iteration and self-correction, unlike Terminus-2’s linear sequence.All evaluated agents also occasionally make operational mistakes such as syntax errors or missing deliverables.Source: §3.4 · p. 6
  • GPT-5.4 tends toward concise solutions or giving up, Claude models keep steadier execution patterns, and Gemini’s external searches can cause information drift.The reported drift can replace an initially correct hypothesis with conflicting online information, notably on knowledge-intensive benchmarks.Source: §3.4 · p. 6

4 Harbor-Index: Compact, Diverse, Challenging, and High-Quality

Harbor-Index curates difficult, diverse tasks through difficulty filtering, AI and human review, and iterative auditing and repair. Its evaluation indicates that the compact suite remains challenging while allowing comparisons of pass rate and run cost across models and harnesses.

4.1 Construction Pipeline

  • The audit funnel reduces 6,627 tasks to 82 across 29 benchmarks, retaining tasks judged well-defined and genuinely difficult after review and repair.Filtering, AI auditing, human review, panel selection, and trajectory-grounded final checks progressively remove broken or too-easy tasks.Source: §4 · p. 6, §4.1 · p. 7
  • Across more than 30 benchmarks, roughly one third of the hardest candidate tasks are rejected as broken rather than genuinely difficult.Examples include GAIA2 simulation events that never fire and verifier requirements not specified by task instructions.Source: §4.1 · p. 7

4.2 Evaluation Settings

  • Evaluation covers nine models under Terminus-2 and a model-native harness, with free-form tasks judged by an LLM and other tasks scored programmatically.Continuous metrics are converted to pass/fail using a threshold requiring outcomes to advance the current frontier.Source: §4.2 · pp. 7–8

4.3 Evaluation Outcome

  • The three highest-scoring models benefit most from vendor-native harnesses, while several open-weight models reach the cost–performance Pareto frontier under Terminus-2.Failures comprise 453 timeouts, 361 near-miss solutions, and 445 fundamentally wrong answers; matched task by task, native harnesses reduce timeout rate from 42% to 26%.Source: §4.3 · pp. 8–9, Figure 6 · p. 8

5 Related Work

Related work covers the diversity of agent benchmarks and infrastructure, benchmark validity and task quality, and methods for efficient evaluation. It describes both the value of broader evaluation and the limits of aggregating or compressing benchmark results.

Agentic benchmarks and evaluation infrastructure

  • Agent benchmarks span many domains and interaction types, but their distinct action spaces, runtime assumptions, and scoring block aggregation.Prior work also finds reported scores depend on interface, scaffold, cost budget, and holdout hygiene [182] [175] [66].Source: §5 · p. 9

Benchmark validity and task quality

  • Task and verifier flaws can destabilize agent evaluations, shift measured performance, and even reorder leaderboard standings.Reported problems include label errors [116], benchmark misdesign shifting performance by up to 100% relative [208], and weak tests [3] [168] [192] [86].Source: §5 · p. 9

Efficient evaluation and benchmark redundancy

  • Low-rank score matrices motivate curated subsets and adaptive item selection, but subset prediction can degrade on stronger models and micro-benchmarks may need more items than assumed.Agreement estimates also depend on unstandardized choices [124].Source: §5 · p. 9

6 Discussion and Conclusion

This section discusses Harbor Adapters and Harbor-Index, reports broad evaluation findings and limitations, and catalogs adapter task setups and validation across benchmark families. It also notes supported benchmarks excluded from experiments for compute- and time-related reasons.

6 Discussion and Conclusion

  • The evaluation covers only 54 collected adapter benchmarks and a focused set of popular models; its findings are not universal across LLMs, agent architectures, or evaluation settings.The authors warn that stronger agents may exploit benchmark artifacts or evaluation loopholes, creating reward hacks and unreliable results.Source: §6 · p. 9
  • Across 54 benchmarks, 16 model–harness configurations, and 6,627 tasks, base-model capability primarily drove performance gaps; many benchmarks had redundant ranking signal, and frontier models were more token-efficient but costlier.The conclusion also reports that low-resolution tasks often reflect task-design flaws rather than genuine difficulty.Source: §6 · p. 9
  • Harbor-Index 1.0 distills the evaluation suite into 82 compact, difficult, diverse, audited tasks spanning 29 benchmarks.The authors plan to maintain it as a live benchmark, updating tasks to reduce saturation and preserve quality while integrating more adapters.Source: §6 · p. 9

D.1 Motivation

  • Harbor Adapters reduce integration effort from O(mn) to O(m + n) by translating heterogeneous benchmarks into a shared interface with sandboxed execution and reward-based evaluation.The motivation is that separate benchmark-agent integrations are costly and inconsistent environments, instructions, tools, and configurations hinder direct leaderboard comparisons.Source: §D.1 · p. 33

D.2 Agent Interaction Modes

  • The adapter catalog classifies benchmarks as agentic, non-agentic, or modified-agentic, with interaction mode determining the applicable integration scenario.Agentic tasks require iterative environment interaction, while non-agentic tasks are single-pass and modified-agentic tasks change measurably when agent capabilities are introduced.Source: §D.2 · p. 33

D.3 Adapter Construction Workflow

  • Adapter development combines schema conversion, environment and evaluator reconstruction, oracle checks, human and automated review, and periodic standardization.Oracle solutions are checked for a 100% pass rate; a five-reviewer team and automated bots support review, while standardization keeps adapters, contributor guidance, and review prompts aligned.Source: §D.3.1 · p. 33, §D.3.2 · p. 34, §D.3.3 · p. 34

D.4.2 Design and Evaluation Metrics

  • Parity verification compares original and Harbor benchmarks under matched conditions, using repeated runs and means within a margin consistent with standard errors.Runs use the same agent, model, tools, prompt, decoding, and execution setup; k = 3 by default, and adapters failing parity are debugged and rerun. Integration may use a supported native agent, a Harbor CLI agent, or a ported custom agent.Source: §D.4.2 · p. 34, §D.4.1 · p. 34, §D.4.3 · pp. 34–35

D.4.4 Parity Set Selection

  • Parity uses full task sets when feasible and documented representative subsets otherwise; the catalog includes 300-task SWE-Bench-Multilingual and 731-task SWE-Bench Pro adapters.Parity subsets include 50 stratified multilingual tasks and 100 random SWE-Bench Pro tasks, followed by a 10-task rerun after adapter fixes.Source: §D.4.4 · p. 35, §D.6.1 · p. 36

D.6.1 Software Engineering

  • SWE-Bench Verified covers 500 Python unit-test tasks; oracle solutions pass 99.2%, and Harbor checks parity across all 500 plus official leaderboard scores on 499.The software-engineering subsection also covers competitive and function-level coding, optimization, and language translation benchmarks.Source: §D.6.1 · pp. 36–38

D.6.2 Mathematics & Reasoning

  • Math adapters span competition and procedural reasoning, from 60 AIME problems with 100% oracle pass to 167 ARC task pairs evaluated under the upstream Pass@2 protocol.AIME lacks an agent-oriented upstream harness, so parity is instead checked by whether diverse agents recover the expected fraction of solutions.Source: §D.6.2 · pp. 38–39

D.6.3 Knowledge & Long Context

  • Knowledge coverage ranges from expert and multilingual QA to 99 adapted AA-LCR long-context tasks; AA-LCR excludes one ground-truth error and its oracle passes 99/99.AA-LCR evaluates reasoning over roughly 100k-token documents across seven categories.Source: §D.6.3 · pp. 39–40

D.6.4 Scientific Research

  • Scientific-research adapters cover replication, scaling-law discovery, scientific code, and biomedical QA; SciCode oracle validation is limited to 15/15 validation problems because test oracles are placeholders.Other examples include numeric-tolerance replication tasks, continuous scaling-law metrics, and notebook-derived biology questions.Source: §D.6.4 · pp. 40–41

D.6.5 Agents, Tools & Systems

  • The agents/tools subsection names GAIA, GAIA2, and BFCL under tool use and assistants, then introduces DeepSynth, Seal-0, and WideSearch as deep-research and web-agent benchmarks.Source: §D.6.5 · p. 41
  • Harbor’s additional agents-and-systems adapters reuse upstream datasets and scoring, with oracle passes of 40/40 for DeepSynth, 111/111 for Seal-0, and 200/200 for WideSearch.WideSearch reports mean Item F1 of 0.9996; its small oracle gap comes from markdown-delimiter conversion in gold-answer cells, which the paper says does not affect real agent runs.Source: §D.6.5 · pp. 41–42

D.6.6 Data & Analytics

  • DA-Code adapts 479 tasks with 11 evaluation-script fixes and no significant parity difference, while Spider 2 adapts 64 tasks and passes its 64/64 oracle.DA-Code excludes the ml-cluster category; Spider 2 excludes four tasks that consistently fail the oracle on both sides.Source: §D.6.6 · p. 42

D.6.7 Professional Domains

  • FinanceAgent, PIXIU, and SpreadsheetBench adapters use benchmark-specific judging or metrics, with all oracle solutions passing for PIXIU and SpreadsheetBench.FinanceAgent covers SEC-filing analysis; PIXIU materializes examples across 29 subcategories, and SpreadsheetBench uses cross-platform LibreOffice recalculation with 11 evaluator fixes.Source: §D.6.7 · p. 42
  • MedAgentBench and LawBench adapters reuse upstream grading and semantics, with oracle passes on all 300 MedAgentBench tasks and the full LawBench task set.MedAgentBench adds strict-payload instruction formatting for FHIR workflows; LawBench uses per-family metrics and a stratified 120-task parity subset.Source: §D.6.7 · pp. 42–43

D.6.8 Safety & Security

  • CyberGym and StrongReject cover security evaluation through vulnerability-triggering inputs and jailbreak-refusal scoring, with validation and parity performed on subsets rather than their full task sets.CyberGym’s primary level matches the upstream paper, but full-set validation is skipped because its runner images occupy roughly 10 TB; StrongReject supports all 39 jailbreak methods and uses inverted scoring, where reward 1 indicates safe refusal.Source: §D.6.8 · p. 43

D.6.9 Multimodal

  • The MMAU audio adapter transcribes audio with Qwen2-Audio-Instruct before running the upstream judge, and its oracle passes the full test-mini set.Parity uses the full 1,000-task test-mini set for three trials.Source: §D.6.9 · p. 43

D.6.10 Benchmarks Using Harbor Format

  • Terminal-Bench, CompileBench, and SkillsBench need no additional adaptation because their tasks are authored in Harbor format and run with upstream test scripts or verifiers.SkillsBench also compares agent runs with and without curated skills exposed through Harbor’s skills mechanism.Source: §D.6.10 · p. 43

D.6.11 Other Benchmarks Not Included

  • The authors exclude supported benchmarks from experiments because of compute and time constraints, although their adapters are merged into Harbor or have approved pull requests.The passage notes that benchmarks with completed parity experiments also appear in Tables 3 and 4.Source: §D.6.11 · pp. 43–44

D.7 Benchmark Issues and Lessons Learned

The adapter audit surfaced recurring reproducibility and validity failures across benchmarks, grouped by symptom and followed by recommendations for benchmark authors. The evaluation protocol specifies how datasets are prepared, sampled, executed, and audited.

D.7 Benchmark Issues and Lessons Learned

  • Unpinned dependencies, broken builds, and platform-specific tools can prevent tasks from running or exceed the default backend’s hardware budget.Examples include porting SpreadsheetBench to LibreOffice and a SWE-Bench Verified task that exceeds the default execution backend’s hardware budget.Source: §D.7 · p. 44
  • Hidden nondeterminism and grader divergences distort scores; GSO’s timing-sensitive metric marks 13 of 102 oracle solutions slower than baseline on some runs.The authors document divergences as benchmark bugs and keep oracle exclusions out-of-band rather than baking them into the dataset.Source: §D.7 · p. 44
  • Undocumented preprocessing, prompt, and grading conventions can change outcomes, including positional bias in LAB-Bench unless answer choices are randomized.Other examples include graders that reject gold patches without a trailing newline and an upstream regex that returns an empty set for nine SWT-Bench tasks.Source: §D.7 · p. 44
  • Gold solutions sometimes fail their own tests: examples include oracle pass rates of 99.2% for SWE-Bench Verified and 87.3% for GSO.Failures often reflect environment drift; without explicit oracle validation, they can remain invisible and inflate downstream baselines.Source: §D.7 · p. 45
  • Agent-hacking surfaces include public-answer exfiltration, exposed gold labels, and shared-environment verifier tampering that can silently inflate scores.The authors flag these surfaces during adapter review and require network restriction, identifier masking, or verifier isolation before merge.Source: §D.7 · p. 45
  • The authors recommend pinned dependencies, reproducible containers, runnable oracle checks, deterministic scoring, separable task specifications, and documented assumptions.They expect these practices to reduce per-benchmark integration cost.Source: §D.7 · pp. 45–46

E Evaluation Protocol and Reproducibility Details

  • The evaluation protocol specifies runnable task registries, locked paper subsets, isolated model–harness execution, and stored trial metadata and artifacts for auditing.The protocol is designed to make reported results reproducible and auditable.Source: E Evaluation Protocol and… · p. 46

E.1 Evaluation Infrastructure

The evaluation infrastructure uses Harbor to prepare and execute benchmark tasks, record trial artifacts, and support quality checks and audit. The paper reports broad benchmark results and analyzes score-matrix structure, including a distinct harness-effect dimension.

E.1 Evaluation Infrastructure

  • Harbor executes isolated trials through Daytona by default, using Docker for incompatible or resource-intensive tasks and Modal for GPU-dependent tasks.The evaluation pins Harbor versions and uses provider-default model settings unless otherwise specified; the authors caution against direct comparison with leaderboards using different settings.Source: §E.1 · p. 46
  • 793,698 public trial records total about 340 GB; Harbor-Index 1.0 runs 9 models across two harness conditions on 82 tasks, for 1,476 runs.The release also provides a lightweight manifest for filtering trial identifiers without downloading trajectory data.Source: §E.1 · p. 47

E.2 Dataset Preparation and Sampling

  • The standard-job-compatible subset contains 54 benchmarks and 6,627 tasks, with smoke tests for execution-platform compatibility and i.i.d. sampling for large benchmarks.The manifest records selected and original task counts, sample rates, verifier types, and execution backends.Source: §E.2 · p. 48, Table 5 · p. 48

E.3 Error Handling and Quality Control

  • Evaluation proceeds from a 1% compatibility and verifier sanity check, to a 10% matrix run, then to full evaluation, with infrastructure-caused trial failures rerun.Agent timeouts within the benchmark envelope, policy refusals, and other behavior-derived failures are not rerun.Source: §E.3 · pp. 48–49

E.4 Harbor-Index Experiment Protocol

  • Harbor-Index 1.0 contains 82 tasks across 29 benchmarks; continuous rewards use locked task-specific pass thresholds, and audit-stage timeouts are tightened.Timeouts are typically set to 1.2 times the fastest frontier model’s observed runtime, or up to three hours when every model fails.Source: §E.4 · p. 49

E.5 Evaluation Results

  • Tables 6, 7, and 8 present benchmark scores for all studied model–harness configurations across the studied benchmarks.The passage identifies the tables’ scope but does not provide their cell values.Source: §E.5 · p. 49

F Detailed Large-scale Quantitative Analysis

  • The quantitative analysis examines cross- and within-benchmark redundancy, model-versus-harness effects, benchmark progress, cost, and execution latency.It analyzes the 16 × 54 score matrix.Source: F Detailed Large-scale Quantitative… · p. 49

F.1.1 Low-rank structure for benchmark space

  • 67.1% PC1 and 10.1% PC2 on all 16 configurations, versus 73.7% PC1 under Terminus-2 alone, indicate a rank-2 space with a distinct harness axis.The first two components capture 77.2% of variance in the full matrix; the figure contrasts the fixed-harness and full matrices.Source: Figure 7 · p. 50, §F.1.1 · p. 50

F.1.2 Benchmark predictability.

The analysis predicts held-out benchmark scores from the remaining score matrix using logit-space regression and low-rank completion, then evaluates prediction error on raw scores. These methods outperform a benchmark-mean baseline, indicating substantial cross-benchmark redundancy.

F.1.2 Benchmark predictability.

  • The evaluation holds out 50% of each model–harness configuration’s benchmark scores across five random seeds and predicts in logit space.Scores are normalized and bounded in [0, 1]; the final predictions are evaluated on raw scores.Source: §F.1.2 · pp. 50–51
  • LogitBenchReg fits univariate regressions from other benchmarks, selects up to five predictors with R2 ≥ 0.1, and combines their estimates for held-out scores.Scores are clipped away from 0 and 1 before logit transformation to avoid infinite logits.Source: §F.1.2 · pp. 50–51
  • SVD-Logit applies iterative low-rank matrix completion to the logit-transformed, column-standardized matrix, replacing held-out entries with rank-2 approximations.Training entries remain observed during the replacement step.Source: §F.1.2 · p. 51
  • When LogitBenchReg produces no prediction, the procedure falls back to SVD-Logit; when both fail, it uses the benchmark-column mean.A regression predictor must pass R2 ≥ 0.1 to produce a prediction.Source: §F.1.2 · p. 51
  • On the 16×54 matrix with 50% holdout and 3 folds, the BenchPress blend achieves MedAE 0.063, a 46% reduction from the column-mean baseline of 0.117, indicating substantial cross-benchmark redundancy.Source: §F.1.2 · p. 51
  • ResearchCodeBench (MedAE = 0.221), FinanceAgent (0.202), and LAB-Bench (0.168) resist prediction, while HumanEvalFix (0.004) and KUMO (0.017) are well reconstructed.Higher MedAE marks benchmarks with less predictable, more distinctive signal in this analysis.Source: §F.1.2 · p. 51, Table 9 · p. 51

F.1.3 How many benchmarks span the evaluation space?

Greedy selection identifies 12 benchmarks with independent signal at the |ρ| < 0.7 threshold, and the informative set spans a broad difficulty range.

F.1.3 How many benchmarks span the evaluation space?

  • 12 of 54 benchmarks meet the independent-signal criterion: each of the first 12 selections has maximum absolute correlation below 0.7 with those already selected.From step 13 onward, every remaining benchmark has |ρ| ≥ 0.7 with at least one selected benchmark.Source: §F.1.3 · p. 52, Table 10 · p. 52
  • About 12 well-chosen benchmarks across domains capture most independent signal in the 54-benchmark suite, according to the authors’ selection analysis.The first redundant benchmark is swebench-verified at step 13, with ρ = 0.70.Source: §F.1.3 · p. 52
  • Informative benchmarks span the difficulty spectrum, from CodePDE at mean score 0.36 to StrongReject at 0.92; uniqueness is not confined to one difficulty level.The passage attributes uniqueness to the distinctiveness of the measured capability axis, not raw task difficulty.Source: §F.1.3 · p. 52

F.1.4 Does the same redundancy hold at the task level?

Task-level analyses show that benchmark scores conceal meaningful variation in task redundancy and that a small set of informative tasks can often approximate broader evaluation. The degree of redundancy and the predictive value of within- versus cross-benchmark tasks vary across benchmarks.

F.1.4 Does the same redundancy hold at the task level?

  • Task-level PC1 explains 32.0% of variance, versus 67.1% at the benchmark level, showing that aggregation compresses independent dimensions.The analysis filters 731 zero-variance tasks from the 6,627-task matrix.Source: §F.1.4 · p. 52
  • Within benchmarks, PC1 explains 49.5% of task variance on average, and a median of 8 components captures 90%.PCA covers 52 benchmarks with at least 10 tasks; zero-variance tasks are removed.Source: §F.1.4 · p. 52
  • Task redundancy differs substantially: HumanEvalFix has PC1 = 86.2% and needs 2 components for 90%, while ResearchCodeBench has PC1 = 78.4% and needs 3.FinanceAgent has PC1 = 69.5% and needs 6 components for 90%.Source: §F.1.4 · p. 52
  • Ranking fidelity from selected tasks ranges from WideSearch (ρ = 0.99) to CyberGym (ρ = 0.75).Table 12 gives the per-benchmark Spearman correlation between rankings from oracle-selected tasks and all tasks.Source: §F.1.4 · p. 52, Table 12 · p. 53
  • In 41 benchmarks, within-benchmark prediction improves over the task-mean baseline, with a median improvement of 24.3%.The largest gains are HumanEvalFix (67.3%) and KUMO (56.7%).Source: §F.1.4 · pp. 53–54
  • Within-benchmark blends outperform cross-benchmark peers in 37/54 cases, while cross-benchmark peers predict better in the other 17.CyberGym (70%) and BIX-Bench (69%) have many independently informative tasks; HumanEvalFix and ResearchCodeBench reach the ρ > 0.7 threshold within the first few selections.Source: §F.1.4 · p. 54

F.2 Model vs. Harness Effect: Methodology

A mixed-effects model separates model and harness effects while accounting for benchmark difficulty. The estimated model effect range is substantially larger than the harness effect range, while benchmark differences account for 0.75 of score variance.

F.2 Model vs. Harness Effect: Methodology

  • The analysis fits a linear mixed-effects model with model and harness as fixed effects and benchmark as a random intercept, estimated by REML.The random intercept controls for benchmark difficulty; coefficients compare levels with the alphabetically first model or harness as reference.Source: §F.2 · p. 54
  • Benchmark differences account for 0.75 of score variance, supporting the benchmark-level random-intercept specification.The reported ICC is 0.75.Source: §F.2 · p. 54, Table 13 · p. 54
  • The model effect range is 0.451 raw score units, versus 0.087 for harnesses, a 5.2× larger range for models.Six of 8 model coefficients and 2 of 4 harness coefficients are significant at p < 0.05.Source: §F.2 · p. 54, Table 13 · p. 54

F.3 Original benchmark data collections and mapping

Historical comparisons show rapid but uneven progress, especially across mathematics and software engineering, while agent latency varies with task difficulty and outcome. The appendix also cautions that differences in benchmark splits and evaluation configurations limit direct comparisons.

F.4 Performance Progress Over Time

  • Historical scores from papers, leaderboards, and technical reports are plotted as observational progress records; hollow diamonds can mark nonidentical evaluation sets.The figures use best reported performance over time, and connected segments for hollow diamonds are reference comparisons rather than strict apples-to-apples continuations.Source: §F.3 · p. 55, §F.4 · p. 56, Figure 8 · p. 56

F.3 Original benchmark data collections and mapping

  • Historical metrics are aligned to Harbor through conversions or matched splits, while seven benchmarks are excluded when no comparable public metric or split is available.The appendix describes benchmark-specific transformations and alignments, including rate conversions, metric inversions, and weighting for the Harbor task mixture.Source: §F.3 · p. 55
  • Only 16 of 54 benchmarks have an overlapping model or harness configuration, so historical results are useful for tracking progress but often not for direct head-to-head comparison.Existing evaluations may use raw models, specialized scaffolds, or configurations unlike the general-purpose harnesses in Harbor.Source: §F.3 · p. 55

F.4 Performance Progress Over Time

  • Mathematics benchmarks approach saturation quickly, while software-engineering progress is strongest on well-specified tasks and weaker on open-ended engineering work.Function-level and competitive-programming tasks often improve rapidly, unlike tasks requiring repository understanding, environment management, and long-horizon debugging.Source: §F.4 · pp. 56–57
  • Knowledge & Long Context and Agents, Tools & Systems have sparser, often slower progress records, reflecting fewer maintained leaderboards and harder apples-to-apples comparisons.Knowledge benchmarks can be expensive and sensitive to retrieval or context-window assumptions; agent evaluations depend on changing environments, APIs, and multi-step protocols.Source: §F.4 · pp. 57–58
  • External best results exceed Harbor results on several benchmarks, in part because they use specialized models or scaffolds and larger agent compute budgets.Harbor instead evaluates a fixed model set under a unified adapter-based setup to improve cross-benchmark comparability.Source: §F.4 · p. 58

F.5 Latency Study

  • Failure latency peaks around difficulty 0.7–0.8 and then declines; GPT-5.4 spends similar time on either outcome, while GPT-5-Nano persists longest on failures.The failure pattern contrasts with the upward success-latency trend; the figure omits extreme difficulty bins where trial counts are low.Source: Figure 10 · p. 58, §F.5 · p. 59
  • Successful-trial latency rises with difficulty, with Claude Sonnet 4.6 and Opus 4.6 in a high-latency group and GPT-5.4, Haiku 4.5, GPT-5-Mini, and Gemini 3 Flash in a low-latency group.The gap between these groups widens on harder tasks.Source: Figure 10 · p. 58, §F.5 · pp. 58–59

G.1 Task Failure

Human review found failures caused by broken environments, lost inputs, mismatches between instructions and verifiers, gameable tests, and incorrect gold answers. These cases show that some failures reflect task design rather than agent capability.

G.1 Task Failure

  • All four reviewed GAIA2 tasks shared a broken environment that never advanced the simulation, causing every model and harness to fail regardless of agent capability.One trajectory ran for 270 steps over 36 hours without receiving anything beyond empty notifications.Source: §G.1 · p. 59
  • The MMAU adapter discards audio payloads, leaving agents with transcripts for tasks graded on acoustic judgments such as emotion or speaker identity.Those tasks cannot be solved as intended regardless of model strength.Source: §G.1 · p. 59
  • Instruction-verifier mismatches arise when tests demand undisclosed formats or files, judges reject correct answers, or task instructions fail to specify the expected fix.Examples include an absent test file, unstated unit formatting, and a bug report that does not direct agents to make a fix.Source: §G.1 · p. 59
  • A FEATUREBENCH-MODAL test failed identically across 14 runs because its undisclosed import path made success depend on guessing a specific code structure.The task therefore measured whether agents guessed the required symbol name and module path, not whether they implemented the feature.Source: §G.1 · p. 59
  • On several AA-LCR tasks, all 16 evaluated configurations reached the same alternative answer, and manual inspection confirmed the gold answer was wrong.The reviewed cases show that unanimous frontier-model disagreement with a gold answer warrants checking the gold.Source: §G.1 · p. 60

G.2.1 Methodology

This section describes how agent failure modes are defined and analyzed, including rubric development, human validation, LLM-judge calibration, and decision rules for classifying failures. It also reports harness-dependent failure patterns and explains how Harbor-Index tasks are sampled, filtered, and audited.

G.2.1 Methodology

  • The taxonomy is bootstrapped from stratified failed trajectories, then independently annotated by two three-person panels and adjudicated with a recall-leaning union rule.Each annotator applies candidate rubrics as MATCH/NO MATCH with trajectory evidence; a cell is gold MATCH if any annotator in either round marks it MATCH.Source: §G.2.1 · p. 60
  • A multi-rubric shortlist prompt outperforms per-rubric calls, while single-rubric edits compose better than multi-rubric edits during calibration.For Agent Timeout, the original short rubric yields κ = 0.48, whereas an elaborate override yields κ = 0.0; the chosen judge is gemini-3.1-pro-preview after 14 tuning rounds.Source: §G.2.1 · pp. 60–61
  • The calibrated judge reaches pooled κ = 0.51 versus the human–human ceiling of 0.66; six rubrics clear κ ≥0.5, with mean κ = 0.61 versus 0.71 for humans on those rubrics.The remaining six are grouped into “Others”; the pooled gap is concentrated there, particularly for Unverified Claim, Wrong Output Schema, and Wrong Target / Layer.Source: §G.2.1 · pp. 60–61
  • The calibrated judge audits 5,898 failed trials, with retries achieving 98.4% parsed coverage; the remaining 1.6% are dropped from headline numbers.Most dropped transcripts exceed the gateway’s per-request token cap.Source: §G.2.1 · p. 61

Syntax / Language Error. TRUSTWORTHY; Algorithmic Bug. TRUSTWORTHY; Wrong Factual Answer. TRUSTWORTHY

Wrong Output Schema. TRUSTWORTHY; Hidden-Test Regression. TRUSTWORTHY; False Success Claim. TRUSTWORTHY

Unverified Claim. TRUSTWORTHY; Plan Over Implementation. TRUSTWORTHY; Silent Deliverable. TRUSTWORTHY

Agent Timeout. TRUSTWORTHY

  • Agent Timeout applies when the trial ends at the wall-clock or step-count limit and may co-occur with content-failure rubrics for problematic partial output.In co-occurring cases, both rubrics should be shortlisted rather than selecting only one.Source: Agent Timeout. TRUSTWORTHY · p. 67
  • Agent Timeout matches when timeout evidence appears in trial data, even if a content rubric also applies; natural completion and verifier-only timeouts do not match.Reviewers inspect error fields, runtime, and the final message, quote the timeout signal, and may shortlist a content co-rubric when partial output also supports it.Source: Agent Timeout. TRUSTWORTHY · p. 67

Environment Block. TRUSTWORTHY

  • Environment Block requires a documented constraint, a good-faith attempt, and no reasonable workaround; a completed numerical pipeline outside tolerance is instead a Calculation Mismatch.Wrong commands or paths, successful workarounds, guesses without a pipeline, crashes, and agent timeouts are excluded from these matches.Source: Environment Block. TRUSTWORTHY · p. 68

Rule Inference Error. EXCLUDED

  • Rule Inference Error requires a demonstrated misapplication of a rule inferred from examples, while Specification–Verification Mismatch requires a defensible deliverable rejected for an unstated or contradictory verifier requirement.Both rubrics default to no match when evidence is uncertain; ordinary Q&A or coding errors do not qualify as rule inference, and wrong content or purely formatting failures do not qualify as specification mismatch.Source: Rule Inference Error. EXCLUDED · pp. 68–69, Specification–Verification Mismatch.… · p. 69

Insufficient Web Research. EXCLUDED

  • Insufficient Web Research applies when required retrieval was feasible but absent, inadequately corroborated, uniformly blocked despite alternatives, or conducted through an improper channel.It does not apply to internal tasks, unavailable search tools, or cases where at least two lookups were reconciled, even if the answer was wrong.Source: Insufficient Web Research. EXCLUDED · pp. 69–70

Premature Termination. EXCLUDED

  • Premature Termination requires a concrete next action narrated but not executed; Wrong Output Location requires content written somewhere other than the required location.A completed narrated action or a timeout does not count as premature termination, while an absent deliverable, wrong format, or wrong content does not count as wrong location.Source: Premature Termination. EXCLUDED · p. 70, Wrong Output Location. EXCLUDED · p. 70

G.2.2 Agent Failure Mode Full Results

  • Environment Block is 9.3% for GPT-5.4 under Terminus-2 versus 3.5% under Codex, while Silent Deliverable peaks at 6.9% on Gemini CLI and 5.0% for Opus 4.6 under Terminus-2.Capability rubrics are broadly conserved across harnesses within model families, while operational rubrics vary by harness; multiple rubrics can fire on one trial, so row percentages need not sum to 100%.Source: Table 18 · p. 71, §G.2.2 · p. 71

H.1 Sampling Protocol

  • The difficulty filter samples 18 trials per task across six model–harness pairs and retains tasks with at most six successes, using benchmark-aware scoring cutoffs.The mix uses Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, each with its native harness and Terminus-2; each pair contributes three valid trials.Source: §H.1 · p. 71

H.2 Task Quality Bar

  • Task-quality review rejects candidates for either test–instruction misalignment or difficulty caused by clerical details rather than substantive reasoning or execution.The same rubric guides the AI auditor and human reviewers; calibration examples help identify analogous flaws, including verifier mismatch, fragile string equality, under-specified formats, and invalid gold outputs.Source: §H.2 · pp. 71–72

H.3 Task Audit Process

Harbor-Index task auditing combines rubric-based AI review with trajectory-grounded human review and repeated repair. The process checks instruction–verifier alignment and genuine difficulty, then removes tasks that remain broken or become too easy.

H.3 Task Audit Process

  • On 74 shared tasks, Opus-4.7 and Gemini-3-Flash agreed on 89.2% of three-way decisions (κ = 0.747); binary accept/non-accept agreement was 93.2%.The authors use Gemini-3-Flash as the production auditor, noting that on this subset it accepted no task that Opus-4.7 rejected.Source: §H.3 · p. 72
  • The task-quality judge checks instruction–test alignment and genuine difficulty, using trial outputs and trajectories to distinguish task defects from agent or infrastructure failures.Difficulty should arise from reasoning, domain expertise, or multi-step execution—not ambiguous schemas or clerical details.Source: §H.3 · pp. 73–74

H.4 Human Review Protocol

  • Fourteen domain-experienced reviewers screen candidates; senior reviewers select 100 for trajectory-grounded audit, repair fixable defects, and remove tasks that remain broken or become too easy, yielding 82.The workflow was iterative rather than a fixed-panel annotation study; reviewer counts vary by task, so the authors report minimum coverage rather than a single human agreement statistic.Source: §H.4 · pp. 74–75

H.5 Audit-and-Fix Loop

  • Repeated automated audits and reviewer repairs check verifier outcomes against trajectories, tests, and reference solutions until the task set stabilizes.The loop prioritizes false positives and false negatives; it also tightens per-task timeouts and uses separate verifier sandboxes to close shared-container reward hacks.Source: §H.5 · p. 75

Harbor-Index 1.0 Task Catalog

  • Table 19 summarizes the 82 release tasks by domain and benchmark; task IDs and interactive per-task scores for all 1,476 rollouts are published.The task-level artifacts are available through the Harbor-Index site and Harbor Hub.Source: Harbor-Index 1.0 Task Catalog · p. 75

I Limitations

The evaluation remains costly, and Harbor-Index is only a proxy for the full benchmark distribution. Automated judges may still be biased or inconsistent, while downstream societal impacts are outside this work’s scope.

I Limitations

  • Large-scale evaluation requires substantial compute and token budgets, which may limit reproducibility for smaller research groups; Harbor-Index remains a proxy for the full benchmark distribution.The authors identify reducing evaluation cost as future work.Source: I Limitations · p. 75
  • LLM-as-a-judge evaluations may retain bias, variance, or systematic errors, especially for open-ended or subjective tasks, despite aggregating four models and repeated runs.Source: I Limitations · p. 75
  • The work does not address downstream societal impacts, and future work should consider safety, alignment, governance, benchmark diversity, adapter verification, and more realistic dynamic protocols.Source: I Limitations · p. 75

J Broader Impacts

Harbor Adapters and Harbor-Index aim to improve the scalability, reproducibility, and coverage of agentic evaluation. The paper describes potential benefits and risks, and recommends interpreting results cautiously and complementing them with domain-specific and real-world testing.

J Broader Impacts

  • The infrastructure is intended to lower barriers to evaluating agents across diverse benchmarks and support more comprehensive, reproducible comparisons.The authors say this may help identify capabilities and failure modes and avoid overfitting to a small set of popular benchmarks.Source: J Broader Impacts · pp. 75–76
  • Standardized evaluation may encourage benchmark overfitting, and improved infrastructure could indirectly support more capable autonomous agents with potential misuse risks.Source: J Broader Impacts · p. 76
  • The authors partially mitigate these risks through Harbor-Index’s compactness, diversity, difficulty, and quality, and by releasing detailed failure-mode analyses.Source: J Broader Impacts · p. 76
  • Users are encouraged to interpret results cautiously, avoid relying on a single benchmark, and add domain-specific and real-world testing; the authors call for continued responsible use and scrutiny.Source: J Broader Impacts · p. 76
Loading 2609.04298v3…