PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

Chanhee Park, Jeongho Yoon, Sungbin Han, Hyeonseok Moon, Heuiseok Lim

arXiv:2609.34428v1cs.CLcs.AI

TL;DR

Agentic benchmarks often rely on single scores that do not diagnose where multi-step agents fail. AgentHop evaluates scientific question answering through a constrained sandbox and four diagnostic axes, finding model-family behavior patterns and per-model differences that aggregate accuracy misses.

  • Problem

    Agentic tasks combine multiple stages, while many benchmarks report a single score and provide limited behavior-decomposed evaluation.

    Source: §1 · p. 1

  • Method

    AgentHop evaluates 1,011 scientific question-answering items in a seven-tool constrained sandbox and decomposes behavior into search, synthesis, tool-use pattern, and resource management.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Across 19 models, behavior clusters by training composition and architecture rather than capability tier, and the four-axis decomposition localizes per-model deficits.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Per-model bottleneck attribution and provider-level trade-offs inform model-selection and trajectory-policy decisions beyond capability tier.

    Source: §5 · p. 9

    Some accuracy may reflect familiarity with cited literature, so the decomposition describes behavior on top of that parametric floor rather than counterfactual agentic capability.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.

1 Introduction

Agentic tasks combine multiple stages, but existing benchmarks often report a single score that obscures where an agent fails. AgentHop introduces a four-axis evaluation framework and a benchmark for diagnosing those behaviors.

1 Introduction

  • Agentic failures can arise at different stages, and developers need to identify the binding stage when selecting models or directing post-training.The stages call for different fixes.Source: §1 · p. 1
  • Existing agent benchmarks largely prioritize single-metric leaderboard comparability, while recent decompositions cover only one or two axes.This leaves cross-axis interactions invisible.Source: §1 · p. 1
  • AgentHop evaluates search, synthesis, tool-use pattern, and resource management in one per-model report to expose sub-ability bottlenecks and cross-axis interactions.The framework pairs one diagnostic axis with each stage of agent operation.Source: §1 · pp. 1–2
  • AgentHop comprises a 1,011-item scientific question-answering benchmark, a seven-tool resource-constrained evaluation, and a four-axis framework applied to 19 models.The evaluation reports per-model strengths and weaknesses, family-level traits, and per-axis diagnosis.Source: §1 · p. 2

2 Related Work

Prior agent benchmarks commonly score end-to-end outcomes or isolate individual capabilities. AgentHop is positioned as a benchmark that combines evidence navigation and synthesis while also diagnosing tool-use patterns and resource management.

Agentic search

  • Agentic-search benchmarks score end-to-end runs, while AgenticRAGTracer localizes errors by hop; AgentHop separates evidence recall from conversion to distinguish retrieval from reasoning.WebArena [18], GAIA [19], and BrowseComp do not separate navigation skill from synthesis at the per-sample level.Source: §2 · p. 2

Knowledge synthesis

  • Knowledge-synthesis benchmarks supply evidence for reasoning in isolation, whereas AgentHop requires agents to locate evidence and then synthesize across it.HotpotQA [16] and 2WikiMultiHopQA focus on multi-hop QA; OpenScholar [7] extends to open-ended scientific synthesis.Source: §2 · p. 2

Tool-use pattern

  • Tool-use benchmarks assess call correctness or task completion; AgentHop adds calls-per-turn and call-sequence fingerprints to characterize how models compose trajectories.BFCL evaluates function-call correctness, τ-bench [22] evaluates end-to-end completion, and TRAJECT-Bench diagnoses trajectory-level call properties.Source: §2 · pp. 2–3

Resource management

  • Resource limits are rarely primary benchmark outcomes; AgentHop treats budgets as part of task difficulty and identifies which cap binds on non-submission failures.AgentBoard [14] decomposes progress without budgeting resources, while SWE-bench [9] treats efficiency as secondary.Source: §2 · p. 3

3 The AgentHop Benchmark

AgentHop places agents in a constrained citation-navigation task, using a constructed scientific QA benchmark and four diagnostic axes to separate retrieval, synthesis, tool-use behavior, and resource use.

3.1 Task Definition

  • Agents receive a seed paper, question, and four answer options, then navigate citations and read evidence using seven sandbox tools under per-run resource limits.Only read_section returns paper content; the run is capped at 20 turns, 200K aggregate tokens, and 30 tool-call budget points, with cap exhaustion before submission scored incorrect.Source: §3.1 · pp. 3–4
  • Items vary by target multiplicity and citation-path depth, yielding four types across single- or multi-target questions and depth-1 or depth-2 paths.The two-hop citation neighborhood averages 311 papers per item.Source: §3.1 · p. 4

3.2 Construction

  • The 1,011-item benchmark draws on 7,205 papers and over 450K citation edges, with single- and multi-target questions grounded in one or two papers.Each item has four options and three engineered distractors; the seed, bridge, and gold papers are guaranteed to have parsed section text.Source: §3.2 · p. 4

3.3 Evaluation Protocol: Four Diagnostic Axes

  • Paper recall measures whether content from every gold paper is read, while section recall requires a delivered section to contain a direct-labelled evidence location.Section labels are mapped to their containing readable section before matching.Source: §3.3 · p. 4
  • Conversion rate is the fraction answered correctly among recalled samples, isolating synthesis from retrieval success.High recall with low conversion indicates located evidence that the model cannot combine; low recall with high conversion points to retrieval as the accuracy ceiling.Source: §3.3 · p. 5
  • Calls per turn indicates sequential or parallel tool use, while token, turn, and budget means and cap-hit rates characterize resource use and non-submission failures.A calls-per-turn value of 1.0 indicates strictly sequential calls; values above 1.0 imply at least one parallel emission.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate 19 models under a shared seven-tool, resource-constrained protocol and decompose their performance across agent behaviors. Results reveal family-specific operating patterns, additional failure mechanisms, robustness evidence, and the relative contribution of search tools.

4.1 Setup

  • The panel comprises 19 models across closed-frontier, open-frontier, and compute-efficient tiers, all evaluated on 1,011 items with seven tools and fixed turn, token, and call-budget limits.The same items were also used for a closed-book, no-tool baseline.Source: §4.1 · p. 5

4.2 Main Results

  • Across 19 models, behavior separates by family rather than capability tier: GPT commits early, Anthropic and GLM verify, several families over-search, and resource strategies diverge.The experiments also distinguish synthesis, tool-use, and resource-management patterns across models.Source: §4.2 · pp. 5–6

4.3 Additional Analysis

  • After wrong-paper reads, Think distinguishes efficient navigators from alternatives: Anthropic checkpoints think on 70–82% of such reads, while floor models submit on 60% or more.Recover is rare across the panel, at no more than 7%.Source: §4.3 · p. 7
  • Non-submissions arise from token-cap, turn-cap, or budget exhaustion, with different model patterns; the authors associate these mechanisms with output-length discipline, commitment training, and cost-aware planning interventions.Capability tier does not predict these failure rates: Gemma-4-26B-A4B loses 44%, compared with 1% for GPT-5.3 codex and 2% for Gemma-4-31B.Source: §4.3 · p. 7
  • Gemma 3→4 and Qwen-dense 3→3.5 improve every axis, while Qwen MoE 3.5→3.6 gains headline accuracy through synthesis but regresses in multi-target paper recall from 0.472 to 0.302.The figure pairs older and newer checkpoints within architectural cohorts and annotates overall-accuracy differences.Source: §4.3 · pp. 7–8, Figure 2 · p. 8

4.4 Robustness Checks

  • AgentHop accuracy correlates more with reasoning and multi-step benchmarks (ρ = 0.67–0.73) than with SWE-Bench Pro/Public (ρ = 0.46) or BrowseComp (ρ = 0.38).The authors characterize AgentHop as measuring multi-paper scientific reasoning rather than a generic capability proxy.Source: §4.4 · p. 8
  • Cronbach’s α is 0.997, and rerun rank orderings are preserved across accuracy and four axes; Gemma-4-31B is an exception in run-to-run variation, with accuracy std 0.031.DeepSeek-chat and GPT-5.3 codex have accuracy standard deviations of 0.009 and 0.006, respectively.Source: §4.4 · pp. 8–9

4.5 Tool Ablations

  • Across three checkpoints, removing keyword search retains most tool-conditioned lift over closed-book (+23 to +35 pp), while adding keyword search contributes only +3 to +10 pp.The ablation table pairs each delta between adjacent settings: zero-tool to no-search, then no-search to full-tool.Source: §4.5 · p. 9, Table 4 · p. 9

5 Discussion

Across 19 models, AgentHop’s four-axis decomposition localizes deficits and reveals family-level behavior that aggregate accuracy cannot recover. These diagnoses inform post-training, model selection, and trajectory-policy decisions.

5 Discussion

  • Across 19 models, agentic behavior separates by training composition and architecture rather than capability tier.The four axes localize deficits across search, synthesis, tool use, and resource discipline.Source: §5 · p. 9
  • Per-model bottleneck attribution identifies which sub-ability is binding, with different scaffolding or training suggested for synthesis-, retrieval-, and resource-bound checkpoints.The discussion links synthesis bottlenecks to cross-document integration scaffolding, retrieval bottlenecks to navigation training, and resource constraints to context-management policy.Source: §5 · p. 9
  • Provider-level tool-call regimes distinguish parallel-versus-sequential training, course-correction after wrong reads, and architecture-clustered failure modes for pipeline decisions.These regime-level findings inform model-selection and trajectory-policy decisions beyond capability tier.Source: §5 · p. 9

A Limitations

AgentHop’s conclusions are bounded by parametric familiarity, a computer-science-only multiple-choice corpus, generator overlap, and uneven diagnostic power; trajectory findings are observational. The paper also describes the dataset, evaluation, annotation, and release practices and recommends pairing scores with target-task benchmarks and per-axis diagnosis.

A Limitations

  • Some accuracy may reflect familiarity with cited literature, and the decomposition does not provide a counterfactual measure of agentic capability.The zero-tool ablation bounds closed-book accuracy at 0.49–0.73 for closed-frontier and 0.18–0.38 for open-frontier checkpoints; the floor is not subtracted.Source: A Limitations · p. 15
  • The corpus covers nine computer-science venues from 2022–2025 and four-option questions, so performance and bottleneck attributions beyond that distribution are unmeasured.The paper makes no claim that per-model bottleneck attributions generalize to other fields, long-tail venues, or open-ended formats.Source: A Limitations · p. 15
  • A disjoint-generator check found GPT-5.4 scores differed by 0.3 percentage points, but its 94-item subset cannot exclude subtler stylistic overlap or support per-axis comparison.Scores were 78.7% on the subset GPT-5.4 did not write and 79.0% on items it generated.Source: A Limitations · p. 15
  • The diagnostic axes have different informativeness across capability tiers, so no axis is claimed to have equal diagnostic power for every model.Paper recall is more discriminating in the middle and bottom of the panel, while conversion rate separates closed-frontier checkpoints more sharply.Source: A Limitations · p. 15
  • Tool-call fingerprints characterize model-plus-interface systems, while trajectory associations are diagnostic correlates and do not establish that tool-use patterns cause accuracy.Provider interfaces differ in reasoning modes, parallel-call support, and tool-call parsing; the authors do not treat strategy–accuracy associations as architectural prescriptions.Source: A Limitations · p. 15

B Ethical Considerations

  • The paper recommends pairing leaderboard scores with target-task benchmarks and consulting per-axis decomposition for failure-mode attribution.It also says benchmark items and gold-section labels should not be incorporated into LLM training corpora.Source: B Ethical Considerations · p. 16

C Construction Pipeline Details

AgentHop’s eight-stage pipeline builds multi-step scientific QA items from citation chains, then filters and audits them for answerability, structural integrity, and evidence quality. The retained benchmark spans a range of difficulty, with depth and target multiplicity marking the clearest difficulty ordering.

Pipeline overview

  • Four generation stages build each sample from seed to four-option MCQ, followed by four filtering stages that retire candidates failing quality criteria.Seeds are expanded into citation chains, Q/A pairs, and distractors before ensemble checks, structural and content sanity checks, human audit, and post-audit triage.Source: C Construction Pipeline Details · p. 16, Table 5 · p. 17

Generation

  • GPT-5.4 generates questions targeting citation evidence or synthesis between cited papers, then adds three diagnostic distractors to each four-option item.Questions avoid naming target-paper methods or titles; distractors answer from the seed, a sibling reference, or no context, and high-overlap options are rejected.Source: C Construction Pipeline Details · pp. 17–18

Automated filtering

  • Ensemble filters remove items answerable from the seed alone or lacking a convergent gold answer with target-paper access.The shortcut check drops 68 samples, while the consensus filter drops 434 weak-gold or otherwise non-convergent cases; the latter also removes factually correct seed-only or no-context distractors.Source: C Construction Pipeline Details · p. 18

Human audit

  • Seven auditors review each Stage 6 survivor on question clarity, gold-answer support, and section-recall labels; 35 of 116 flagged samples are retired after triage.Auditors review 1,078 samples, of which 962 receive Pass verdicts; the largest retirement theme is paper-content extraction failures missed by structural checks.Source: C Construction Pipeline Details · p. 19, Table 6 · p. 19

Post-filtering

  • Post-audit filtering retires five samples for unrecoverable recall labels and 27 for parser-collapsed sections exceeding the 70,015-character cutoff.The length cutoff is calibrated to preserve legitimate long sections, including dense methodology sections.Source: C Construction Pipeline Details · p. 21

Sample-difficulty distribution

  • Single-target depth-1 items are easiest (58.9% high-coverage), while multi-target depth-2 items are hardest (21.3% high-coverage).Across the benchmark, 44.8% of items fall in the mid-coverage bucket and 31.6% in the high-coverage bucket; only 0.2% are solved by every model and 1.1% by none.Source: C Construction Pipeline Details · pp. 21–22, Figure 4 · p. 22

D Experimental Configuration

The evaluation documents model identifiers, serving backends, and model-specific sampling and reasoning settings. It also reports evaluation costs and token consumption across API-served models.

Model identifiers

  • Table 7 maps display names to the API identifiers or Hugging Face paths used, including DeepSeek aliases that resolve to V3.2 during the evaluation window.Reproducing those results requires V3.2 weights rather than the older V3/R1 weights previously associated with the aliases.Source: Table 7 · p. 22, D Experimental Configuration · p. 22

Backends and configurations

  • Closed-frontier models use their providers’ APIs, while open-weight Qwen and Gemma checkpoints run locally on paired H100 GPUs.Sampling temperatures and reasoning configurations vary by model family; for example, Claude Opus 4.6 uses an 8,192-token extended-thinking budget.Source: D Experimental Configuration · pp. 22–23

Cost accounting

  • The full-tool evaluation pass costs approximately $1,294 across 11 API models, and aggregate project API spend is approximately $2.5K.DeepSeek-chat averages 154K input tokens per sample versus 67K for GPT-5.3-codex; eight locally served Qwen and Gemma checkpoints are excluded from API cost accounting.Source: D Experimental Configuration · p. 23, Table 8 · p. 23

E Diagnostic Detail and Robustness Checks

This appendix extends the main experimental analysis through four linked groups of diagnostic checks.

E.1

E.2

E.3–E.4

E.1 Per-axis decomposition (extends Section 4.2)

The appendix expands the diagnostic analyses with stopping behavior, per-cell accuracy, reasoning-type results, and evidence-contact checks. Across models, retrieval success is usually closely associated with accuracy, while some profiles diverge at synthesis.

Per-model stopping behavior

  • Stopping behavior distinguishes premature commitment, continued exploration, and immediate submission, thinking, or further reading across multi-target and single-target scopes.Table 9 reports the full-panel decomposition for MT-fail, ST, and MT-both trajectories.Source: §E.1 · pp. 23–24, Table 9 · p. 24

Per-cell decomposition

  • Table 10 compares per-cell accuracy across single- versus multi-target questions and depth-1 versus depth-2 paths for representative tier checkpoints.Its reported patterns include GPT-5.3 codex staying within 17 percentage points across cells and DeepSeek-chat collapsing on multi-target depth-2.Source: Table 10 · p. 24, §E.1 · p. 24

Reasoning-type breakdown

  • Claude Sonnet’s METHOD accuracy is 83.9% and its MOTIVE accuracy is 71.7%, a 12-point drop and the largest within-model spread in the panel.Across the panel, reasoning-type accuracy ranges only from 53.1% on MOTIVE to 57.1% on METHOD.Source: §E.1 · pp. 24–25

Section-recall label mapping

  • 85.7% of recall-credited trajectories verifiably received direct-labelled evidence text, and conversion separates the evidence-contact groups by roughly 25 points.The decomposition covers 11,072 recall-credited trajectories pooled across the 19-model panel.Source: §E.1 · p. 25, Table 11 · p. 25

Paired conversion tests

  • Exact McNemar tests compare conversion on common recalled items, with cap correction, across leading multi-target scores and tier leader–runner-up pairs.Table 12 reports discordant pairs and two-sided p-values; conversion-tier leaders are shaded only when the corresponding test rejects at p < 0.05.Source: Table 12 · p. 25, §E.1 · pp. 25–26

Joint view: accuracy vs. paper recall

  • Most models cluster near the accuracy–paper-recall diagonal; models above it add synthesis lift, while those below retrieve correctly but lose accuracy at synthesis.Gemini-3 Pro, GPT-5.3 codex, and Gemma-4-31B are above the diagonal; Kimi-K2.5, MiniMax-M2.7, the Qwen MoE family, and DeepSeek-chat are below it.Source: §E.1 · p. 25, Figure 5 · p. 26

E.2 Trajectory and failure analysis (extends Section 4.3)

Trajectory analysis identifies recurring tool-use patterns, their associations with accuracy, and model-specific failure signatures. These patterns reveal differences that aggregate accuracy alone does not show.

Bigram accuracy associations

  • Deliberation transitions are uniformly productive; scaffolded exploration adds +7 to +8 pp, while premature metadata commits lose –10 to –15 pp.Repeated reads and shortcuts into content without deliberation or scaffolding are also identified as anti-productive patterns.Source: §E.2 · p. 26, Table 13 · p. 27

Per-family trigram fingerprints

  • Among closed-frontier models, Opus 4.6 alone has rs→th→sub as its top trigram; 4 of 5 open-frontier checkpoints share ls→rs→rs.In the compute-efficient tier, Gemma-3-27B and Qwen3-30B-A3B concentrate on a metadata-only loop, while Gemma-4-26B-A4B concentrates on greedy reads.Source: Table 14 · p. 27, §E.2 · p. 28

Early-submit-after-wrong-paper signature

  • Submitting within three actions of reading a non-gold paper associates with a −21.1 pp average accuracy difference and is significant in 12 of 19 models.For Claude Sonnet, Claude Opus, Gemma-4-31B, and GLM-5.1, the per-instance penalty is at least 40 percentage points; where the behavior is endemic, broader navigation failure remains.Source: §E.2 · p. 28

Per-model tool-call transition matrices

  • Across all 1,011 samples, five tool-transition archetypes recur per model, align with family stopping signatures, and suggest trajectory shape is a stable training-artifact fingerprint.The patterns include verify-before-commit, confident-commit, and over-search signatures; metadata-only loops occur in legacy compute-efficient checkpoints.Source: §E.2 · p. 28, Figure 6 · p. 29

Non-submission terminations

  • Non-submission caps cluster by architecture: token overflow, max-turns, and budget exhaustion dominate different checkpoint groups.Gemma-4-26B-A4B exceeds the token cap on more than a third of trajectories; parallel-tool regimes nearly exclusively account for budget exhaustion.Source: §E.2 · pp. 28–29

E.3 Robustness and validity (extends Section 4.4)

The section checks AgentHop’s relationship to external benchmark scores and the stability of its measurements across repeated runs. External-score correlations are positive, while run-to-run variability differs by checkpoint.

Cross-benchmark scores and correlations

  • AgentHop accuracy correlated with GPQA-Diamond (ρ = +0.73, n = 15), HLE no-tools (ρ = +0.72, n = 14), Terminal-Bench 2.0 (ρ = +0.70, n = 15), NL2Repo (ρ = +0.70, n = 10), and HLE with tools (ρ = +0.67, n = 10).The external comparison covers a 16-model overlap across seven benchmarks; three lowest-tier compute-efficient checkpoints lack public scores on most benchmarks.Source: §E.3 · p. 29, Table 16 · p. 30

Run-to-run robustness

  • DeepSeek-chat and GPT-5.3 codex stayed within 1 pp of accuracy across three runs (std ≤0.009), while Gemma-4-31B had accuracy std 0.031.Diagnostic-axis rankings were preserved across runs; Gemma-4-31B also had a 21.8K-token spread per trajectory.Source: §E.3 · p. 30

E.4 Tool ablations (extends Section 4.5)

The ablations compare closed-book performance with tool-assisted evaluation and examine when the agent loop helps or hurts. The appendix also specifies the prompt materials and tool-budget design.

E.4 Tool ablations (extends Section 4.5)

  • Weaker closed-book performers that produce valid answers gain more from tool access, while stronger closed-frontier checkpoints gain less.DeepSeek-chat has the panel’s largest absolute lift and ends within a few percentage points of Gemini-3 Pro’s headline accuracy.Source: §E.4 · pp. 30–31
  • Ten of 19 checkpoints score below the 25% four-option chance baseline without tools, with commits attributed to susceptibility to engineered distractors.Interpreting a low closed-book score requires reading it alongside section recall to distinguish weak parametric coverage from full-tool behavioral failure.Source: §E.4 · p. 31, Table 18 · p. 32
  • No exposure stratum shows a monotone closed-book accuracy gradient for the closed-frontier tier; open and compute-efficient tiers sit near the 25% chance floor.The latter tiers’ strata are reported for completeness rather than comparison.Source: §E.4 · pp. 31–32, Table 19 · p. 32
  • Three checkpoints have negative tool-conditioned ∆: the seven-tool sandbox lowers accuracy below closed-book performance through token-cap, non-submission, or metadata-loop failures.Gemini-3 Flash hits the 200K-token cap on nearly a third of trajectories; Gemma-4-26B-A4B’s non-submission rate exceeds 40%.Source: §E.4 · p. 31, Table 18 · p. 32
  • The ablation protocol removes only search_papers for the no-search variant and all tools for zero-tool; cost limits restrict these ablations to three checkpoints.The cohort spans the accuracy range, and the authors do not extrapolate its per-checkpoint deltas to the full panel.Source: §E.4 · p. 32

F Prompts and Tool Catalogue

  • The appendix collects construction and evaluation prompts alongside the evaluation-time tool catalogue; dynamic budget templates and provider wrappers are omitted.Those omitted materials are available in the released harness.Source: F Prompts and Tool Catalogue · p. 32

F.1 Tool catalogue

  • The 30-point budget charges one-point navigation calls and five-point reading calls, while think and submit_answer are free.The cost asymmetry is intended to encourage broad exploration while requiring selective reading.Source: §F.1 · p. 33, Table 20 · p. 33

F.2 Error handling and budget enforcement

This section describes fixed resource limits, tool-use guidance, and error handling, distinguishing retryable errors, malformed tool calls, and hard sample failures. It also presents a same-question contrast between successful and failed agent trajectories.

F.2 Error handling and budget enforcement

  • A 30-point budget rejects unaffordable calls without deduction; consecutive rejections terminate the trajectory, while a successful call resets the counter and exhaustion prompts submission.Agents see the total budget in the system prompt and current resource status in each tool result; when budget reaches zero, the next input adds a soft submit-answer nudge.Source: §F.2 · p. 33
  • The harness terminates trajectories at 200K cumulative tokens or after 20 assistant turns without submission; both limits have explicit termination labels.Token usage is checked after every assistant response; the live turn count appears in the per-result status footer.Source: §F.2 · p. 33
  • Five outcomes classify trajectories: answer, budget, token_limit, max_turns, or error; error follows five consecutive non-transient API failures and is rare in practice.The first four are expected submission or non-submission categories; error is reserved for systematic-failure review.Source: §F.2 · p. 34
  • Hard errors allow up to two sample retries, transient errors retry in place, and malformed tool calls return an error message without retrying the sample.Hard errors include authentication failure, persistent provider 5xx, or five consecutive non-transient API errors within one attempt; transient retries consume neither turn budget nor sample-retry counter.Source: §F.2 · p. 34
  • The run ends at 30 budget points, 20 conversational turns, or 200,000 cumulative prompt and completion tokens; navigation costs 1 point and read_section costs 5.After each tool result, a status line reports turn, token, and budget counters for the agent to track.Source: §F.2 · p. 43
  • Agents are advised to use navigation tools to locate relevant papers and sections, and spend 5-point read_section calls only when they have reason to expect relevant evidence.Source: §F.2 · p. 43

G Example Trajectories

  • The example section contrasts successful Claude Opus 4.6 and failed Kimi-K2.5 trajectories on the same AgentHop sample, mt_0785, a multi-target depth-2 item.Source: G Example Trajectories · p. 44

G.1 The sample

A sample item asks agents to connect evidence across cited papers, and contrasting trajectories show how the diagnostic columns distinguish a successful run from retrieval and budget-related failure.

G.1 The sample

  • The gold answer combines WebGPT’s behavior cloning, reward modelling, PPO, best-of-n, and 56% preference result with SayCan’s affordance/value scoring and 84% planning/74% execution success.Distractors represent parametric reliance, failure to leave the seed paper, or retrieval from the wrong paper.Source: §G.1 · pp. 44–45

G.2 Successful trajectory: Claude Opus 4.6 (8 turns, 19/30 budget)

  • Opus reaches both target papers, reads the relevant WebGPT methods and SayCan evaluation sections, confirms WebGPT’s result, and answers correctly using 19/30 budget points in 8 turns.Its reads are each paired with a think call.Source: §G.2 · p. 45

G.3 Failed trajectory: Kimi-K2.5 (14 turns, 27/30 budget)

  • Kimi spends early turns rereading the seed, then misses SayCan’s evaluation section; an insufficient-budget rejection leaves it with 3 points and it submits the wrong distractor.It used 27/30 budget points across 14 turns.Source: §G.3 · p. 46

G.4 What the diagnostic columns record

  • The diagnostic columns separate Kimi’s correct paper recall from missed section retrieval, while Opus scores correctly on accuracy, paper recall, section recall, and conversion.Aggregate accuracy alone marks Kimi wrong without locating the section-level retrieval miss and subsequent parametric distractor.Source: §G.4 · p. 46

Related papers

Loading 2609.34428v1…