PaperlayerView exact revision on arXiv

Source-linked AI summary

EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks

Mukul Singh, Mansi Uniyal, Devin Devlin, Wen Xie, Big Thadawasin, Ritam Dutt, Vivian Lai, Hyeonsu B. Kang

arXiv:2609.31906v1cs.AI

TL;DR

Email agents must coordinate structured retrieval and state changes across multi-step workflows, where valid calls can still select wrong records or stop early. EmailBench pairs a typed API and linked synthetic corpus with hybrid checks; its evaluations show that successful operations can coexist with incomplete tasks.

  • Problem

    Email workflows require agents to connect structured records and coordinate dependent actions, but valid calls may still target wrong records or leave the workflow unfinished.

    Source: §1 · p. 1

  • Method

    EmailBench evaluates email workflows with a typed API, linked synthetic corpus, scenarios, and complementary state-based and semantic checks.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    94.7% of Sonnet’s scenarios contain at least one tool call and no observed API failure, yet only 33.5% pass task-level grading.

    Source: §6 · p. 8

  • Takeaways & Limitations

    EmailBench provides a concrete setting for studying the gap between local operation execution and end-to-end email-task completion.

    Source: §7 · p. 8

    The evaluated setup uses one persona corpus and omits some scenario-required operations, so outcomes on affected tasks also reflect tool-access, corpus, and environment constraints.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Enterprise email agents must combine information retrieval, structured state changes, temporal reasoning, and multi-step coordination. Recent agent benchmarks include productivity tasks, but few center on typed email workflows in a self-contained environment. We introduce EmailBench, a benchmark of 206 email and productivity scenarios across 16 task categories. The benchmark couples a typed email API specification with provider-neutral naming, a deterministic synthetic Enron-inspired corpus, and a scenario suite whose topic selection was informed by aggregate task-intent telemetry from an interactive prototype. Its hybrid evaluation protocol combines 258 executable static assertions with 211 LLM rubrics. We evaluate eight LM configurations on a fixed single-user corpus. The best-performing configuration passes only 33.5% of scenarios despite 99.7% of its tool calls completing without an observed API failure, with pass rates varying substantially across task categories. This gap shows that valid tool execution is not equivalent to task completion. EmailBench provides a self-contained environment for end-to-end email-agent evaluation, with broader tool coverage, multi-persona testing, and repeated-run evaluation as future work areas.

1 Introduction

EmailBench targets email workflows in which agents must retrieve and connect structured records, make state changes, and coordinate dependent actions. It combines a self-contained benchmark and hybrid evaluation with an empirical analysis of workflow completion.

1 Introduction

  • A workflow may require message retrieval, participant resolution, availability checks, event creation, and follow-up actions; valid calls can still select wrong records or stop early.Later calls may depend on identifiers returned by earlier operations.Source: §1 · p. 1
  • EmailBench pairs a provider-neutral typed email API with a deterministic synthetic Enron-inspired corpus and 206 scenarios across 16 categories.Aggregate task-intent frequencies from an interactive prototype informed topic selection.Source: §1 · p. 1
  • Its hybrid protocol combines executable state and answer checks with scenario-specific LLM rubrics, using score aggregation and critical-check requirements.Source: §1 · p. 1
  • 33.5% is the highest observed task pass rate across eight LM configurations, despite high API-level execution success.Category-level comparisons and recorded trajectories show that executing individual operations does not establish workflow completion.Source: §1 · p. 1
  • The benchmark emphasizes workflow completion rather than API execution alone.Source: §1 · p. 1

2 Related Work

Prior agent benchmarks cover diverse interactive settings, while productivity benchmarks include workflows across office applications. EmailBench focuses on coordinated actions across linked email and productivity records.

2 Related Work

  • Agent evaluation spans heterogeneous environments, web tasks, desktop interaction, and repository-level software engineering, with success tied to externally evaluated outcomes.The cited examples include Liu et al. (2024), Zhou et al. (2024), Yoran et al. (2024), Xie et al. (2023), and Jimenez et al. (2024).Source: §2 · p. 2
  • Tool-use benchmarks emphasize choosing and invoking APIs, whereas office-automation and simulated-app benchmarks place operations within broader workflows.The cited examples include Qin et al. (2024), Patil et al. (2024), Wang et al. (2024), and Trivedi et al. (2024).Source: §2 · p. 2
  • Recent enterprise and productivity benchmarks address cross-functional coordination, long-horizon execution, and stateful services; email and calendar tasks already appear in this literature.EmailBench’s narrower focus is the relationships among mailbox, calendar, contact, and task entities (Wang et al., 2025; Li et al., 2026).Source: §2 · p. 2
  • EmailBench shifts email evaluation from message-content understanding to actions on linked productivity records, combining a typed API, fixed synthetic corpus, scenarios, executable checks, and semantic rubrics.Its combination supports evaluating individual operations and end-to-end task completion, rather than introducing stateful or semantic evaluation in isolation.Source: §2 · p. 2

3 EmailBench

EmailBench combines a typed email API, a linked synthetic corpus, a scenario suite, and a hybrid grader. Its design covers cross-entity workflows and evaluates task completion with executable checks and LLM rubrics.

3 EmailBench

  • EmailBench comprises an API specification, deterministic corpus, scenario suite, and hybrid grader, while the empirical study evaluates only a subset of the specification.The evaluated pipeline runs from scenario query through a restricted adapter to two grading components.Source: §3 · p. 2, Table 1 · p. 2

3.1 API and Corpus

  • The EmailBenchAPI specification defines 119 operational methods across ten domain objects plus identity, with 36 provider-neutral interfaces and typed inputs and outputs.The synthetic corpus links messages to contacts, events, tasks, and meetings; the evaluation backend keeps a separate action log, while message reads remain tied to the original corpus.Source: §3.1 · pp. 2–3

3.2 Scenario Construction

  • The suite contains 206 scenarios across 16 categories, informed by aggregate prototype intent telemetry and expanded through four construction batches.There are 122 scenarios with one declared domain and 84 with two or more; difficulty labels are metadata, not experimentally calibrated levels. Tasks include retrieval, aggregation, state mutation, and multi-step workflows.Source: §3.2 · p. 3, Table 2 · p. 3

3.3 Evaluation Protocol

  • Evaluation combines executable static assertions with scenario-specific LLM rubrics; the suite has 258 static assertions and 211 rubrics, while 43 scenarios have no static assertions.Static checks inspect selected snapshot or answer conditions, while rubrics assess requirements such as completeness and ordering. A scenario passes at score s ≥ 0.5 only if every critical assertion passes.Source: §3.3 · pp. 3–4

4 Experimental Setup

The study evaluates eight LM configurations once each on the same 206 scenarios under a shared tool-calling protocol. All configurations use the same restricted tools and fixed persona corpus, and two additional runs are reported separately.

4 Experimental Setup

  • Eight LM configurations were evaluated once on all 206 scenarios using a shared ReAct-style protocol, a maximum of 15 iterations, and prompts without demonstrations.The configurations were Claude Sonnet 4.5, Claude Opus 4.1, o3, GPT-5, GPT-5.2, GPT-5.4, GPT-5.4-reasoning, and GPT-5.5-reasoning.Source: §4 · p. 4
  • All configurations received the same 46 JSON-schema tools and were evaluated on a fixed synthetic corpus centered on the Vince Kaminski persona.The exposed tool surface excludes boards, directory, and contact-group operations; results include a primary full-suite evaluation and a post hoc access-screened sensitivity analysis.Source: §4 · p. 4
  • The LLM judge scored scenario-specific rubrics from the answer and abbreviated tool trace, and two additional configurations achieved 0% pass rates without tool invocations.The judge shares an endpoint with the configuration labeled GPT-5, leaving possible self-evaluation bias unmeasured; Phi-4 and Mistral Large are reported separately from the eight-configuration comparisons.Source: §4 · p. 4

5 Results

Across eight configurations, task success varies by category and is distinct from successful API execution. The results also examine tool engagement, shared successes, and latency tradeoffs.

5.1 Overall Performance

  • Sonnet 4.5 led the one-run full-suite baseline at 33.5%, ahead of Opus 4.1 at 29.6%; GPT-5 variants ranged from 20.9% to 28.2%.Each configuration was evaluated once, so the observed ranking does not establish repeated-run reliability.Source: §5.1 · p. 4, Table 3 · p. 5

5.2 Category and Composition Effects

  • Sonnet’s category pass rates range from 88.9% on folder tasks to 16.7% on calendar-write tasks, while author-assigned difficulty tiers show no monotonic relationship with pass rates.Pass rates also decline as declared domain count increases: Sonnet falls from 37.7% on one-domain tasks to 14.3% on four-domain tasks; the cohorts differ in task composition.Source: §5.2 · p. 5, Table 4 · p. 5, Table 5 · p. 5

5.3 Execution is not task completion

  • 99.7% of Sonnet’s calls and 99.7% of Opus’s calls completed without observed API failure, yet scenario pass rates remained between 20.9% and 33.5%.A successful call can still select the wrong entities, apply an incomplete filter, or perform only part of a required workflow.Source: §5.3 · pp. 5–6
  • Inbox triage exhausted the 15-iteration limit despite six passing static checks; its critical LLM rubric failed, producing an aggregate score of 0.50.By contrast, the folder-routing example completed both static and LLM grading components.Source: §5.3 · p. 6, Table 6 · p. 6

5.4 Engagement and Completion Timing

  • GPT-5 and GPT-5.2 each had 71 zero-tool failures, compared with seven for Sonnet and four for Opus; retrospective timing curves show when eventual passes finished.By iteration five, 89.9% of Sonnet’s passes and 65.2% of o3’s had finished; these curves are not reduced-budget reruns.Source: §5.4 · p. 6, Table 7 · p. 7, Figure 2 · p. 7

5.6 Latency and Task Success

  • Sonnet paired the highest observed pass rate (33.5%) with a mean latency of 27.06 seconds; several GPT-5 configurations were faster at lower pass rates.The preferred configuration depends on the relative importance assigned to task success and response time.Source: Figure 3 · p. 7, §5.6 · p. 8

5.5 Shared & Configuration-Specific Outcomes

  • Within the selected four configurations, 24 scenarios passed for all, 117 for none, and their combined pass sets covered 89 scenarios.Sonnet–Opus had a Jaccard similarity of 0.711, versus 0.530 for Sonnet–GPT-5.4-reasoning and 0.420 for Sonnet–o3. Calendar, multi-domain, and D3 tasks were overrepresented among shared failures.Source: §5.5 · p. 7

6 Discussion

EmailBench’s results show that successful calls and individual checks can coexist with an unfinished workflow, motivating evaluation of whether requirements collectively cover the request. The discussion proposes progress- and completion-focused interventions as future experiments, while the artifact description documents available materials and provenance limits.

6 Discussion

  • 94.7% of Sonnet scenarios contain a tool call with no observed API failure, yet only 33.5% pass task-level grading.The inbox-triage example also passes all six static checks while failing the critical completeness rubric.Source: §6 · p. 8
  • Call-level and targeted checks do not establish that requirements collectively cover a request, and the hybrid protocol lacks demonstrated agreement with human judgment.Expert calibration and targeted grader comparisons are needed to assess whether the checks capture task completion.Source: §6 · p. 8
  • Future experiments could track progress toward the requested outcome through dependency tracking and checks for unmet requirements before an agent stops.These are hypotheses, not mechanisms validated by the results; proposed experiments should use task-appropriate tools and persona data, intended state semantics, repeated runs, and empirically calibrated difficulty labels.Source: §6 · p. 8

6.1 Reproducibility and Availability

  • The artifact includes the implementation, seed corpus, scenarios, checks, rubrics, and evaluation adapter, but records lack a complete source-commit and tool-schema manifest.A repository-wide license has not been identified, so unrestricted reuse or redistribution should not be assumed.Source: §6.1 · p. 8

7 Conclusion

EmailBench is a self-contained benchmark for enterprise email and productivity agents, evaluated across eight configurations in a single-user setting. Its results show a gap between executing operations and completing workflows, supporting task-centered evaluation while motivating broader evaluation settings.

7 Conclusion

  • Across eight configurations in the evaluated single-user setting, the highest observed task pass rate is 33.5%, despite high API-level execution success.Category-level results and recorded trajectories show that executing individual operations does not establish completion of the requested workflow.Source: §7 · p. 8
  • EmailBench provides a concrete setting for studying the gap between local execution evidence and end-to-end email-task outcomes.The benchmark is self-contained and focused on enterprise email and productivity agents.Source: §7 · p. 8
  • The paper’s supported task-centered standard defines success as completing the user’s requested workflow, not merely executing individual operations.Broader tool coverage, multi-persona testing, and repeated-run evaluation remain future work areas, as stated in the abstract.Source: §7 · p. 8

Limitations

The reported results are bounded by a simulated, fixed-corpus setting, limitations in scenario access and grading, and single-run comparisons that do not isolate model behavior.

Limitations

  • The simulated environment excludes operational conditions such as network failures, rate limits, eventual consistency, and authentication.Its 65-message corpus does not reproduce production-mailbox retrieval and pagination demands; content and queries are English, and tasks use one request rather than interactive clarification or correction.Source: Limitations · p. 9
  • The evaluated setup omits some scenario requirements, uses one persona corpus, and has read and calendar behaviors that can affect task outcomes.Outcomes on affected tasks therefore combine agent behavior with tool-access, corpus, and environment constraints; access-screening does not resolve other environment or grading issues.Source: Limitations · p. 9
  • The hybrid grader is not an exhaustive or independently validated measure of task correctness.Some scenarios rely entirely on LLM rubrics, and the study reports no independent expert calibration or alternate-judge evaluation; checks and abbreviated traces can also limit verification or penalize answers for other reasons.Source: Limitations · p. 9
  • Each configuration has one recorded run per scenario, so variability across executions or judgments is not estimated.Category, domain-count, and difficulty comparisons involve different task compositions and sometimes small cohorts, and do not isolate causal effects.Source: Limitations · p. 9
  • The reported results characterize the evaluated agent, environment, and grading protocol together, rather than isolating model behavior from access and grading constraints.Source: Limitations · p. 9

Ethical Considerations

The paper describes the benchmark’s ethical boundaries alongside supplementary materials on its API and synthetic corpus. The corpus fabricates email content while retaining some real-person identifiers, and live-system actions would carry consequences absent from the simulation.

Ethical Considerations

  • The synthetic benchmark retains names and person-linked contact details, so synthetic content should not be treated as anonymized or as representing named individuals’ conduct.The available artifacts do not establish how prototype intent telemetry was collected or de-identified, and unrestricted reuse rights are not established.Source: Ethical Considerations · p. 9
  • The simulated actions do not certify models for unsupervised live deployment, where messages, records, and meetings can have real consequences.Live adaptation calls for authorization, confirmation of consequential actions, audit logging, recovery mechanisms, and evaluation of safeguards.Source: Ethical Considerations · p. 9

A Supplementary Benchmark Design

  • The appendix documents the API inventory, corpus construction, scenario metadata, grading implementation, tool surface, prompts, examples, and supplementary analyses.Source: A Supplementary Benchmark Design · p. 12

A.1 Provider-Neutral API Specification

  • The API specifies 119 operational methods across ten domain objects plus identity, with provider-neutral typed contracts and snapshots for post-hoc checks.The two implementations are an in-memory mutable API and a changelog API that reads from a frozen corpus and records writes.Source: §A.1 · p. 12

A.2 Corpus Construction and Adaptation

  • The corpus uses Enron themes and relationships as scaffolding, but generates new message content rather than importing original email bodies.The source material includes publicly documented names, roles, addresses, and organizational relationships.Source: §A.2 · pp. 12–13
  • Synthetic calendar events, contacts, todo tasks, board tasks, a mail filter, and directory entries extend the email-only source into productivity entities.The added entities are grounded in message themes, such as events linked to email-mentioned meetings and tasks linked to projects in threads.Source: §A.2 · p. 13
  • Cross-references among entities enable multi-domain scenarios, with the appendix tables summarizing entity counts and interconnection types.Source: §A.2 · p. 13, Table 10 · p. 13, Table 9 · p. 13

A.3 Benchmark Construction Methodology

EmailBench combines a telemetry-informed, synthetic email-and-productivity scenario suite with typed APIs and hybrid grading. In the recorded evaluation, the highest observed pass rate was 33.5%, despite high API-level execution success.

A.3 Benchmark Construction Methodology

  • Aggregate intent-frequency counts from an Outlook-agent prototype informed the 16-category task inventory, while four construction batches added failure cases, domain coverage, personas, and category coverage.The aggregate counts do not establish that the benchmark distribution matches production use.Source: §A.3 · p. 13

A.4 Task Taxonomy

  • Scenarios are organized by capability, declared API-domain count, and author-assigned difficulty; domain counts have a median of one overall and two for D3 tasks.D3 is a qualitative difficulty tier, not a requirement to use three or more domains.Source: §A.4 · p. 14, Figure 4 · p. 14

A.5 Scenario Design

  • The 206 scenarios cover retrieval and analysis, state mutations, and cross-domain workflows, with 16 thematic categories and overlapping task groups.Cross-domain categories test correlating data across API domains and executing multi-step pipelines.Source: §A.5 · p. 14, Table 12 · p. 14

A.6 Hybrid Grading Pipeline

  • The hybrid grading protocol combines executable static assertions with scenario-specific LLM rubrics, using either or both components for each scenario.The suite includes 258 static assertions across four types and 211 LLM rubrics scored across seven semantic dimensions.Source: §A.6 · p. 14, §A.6.1 · p. 14, §A.6.2 · pp. 14–15

A.6.3 Scoring

  • When both grading components are present, their scores receive equal weight; a scenario passes only if its score reaches 0.5 and every critical assertion passes.When only one component is present, that component's score is used directly.Source: §A.6.3 · p. 15

A.7 Grading Inventory and Validation Limits

  • The executed return-contains checker uses literal substring matching even where some scenario patterns are written as regular expressions, and reported grades follow those executed semantics.The available artifacts also lack expert annotations for estimating inter-rater agreement or conducting a grader ablation.Source: §A.7 · p. 15

A.3 Benchmark Construction Methodology

  • The evaluation uses eight recorded configurations on a fixed single-user corpus, and its available artifacts do not include per-scenario records supporting independent manual validation.The adapter omits board, directory, and contact-group tools; the results therefore describe this restricted setup.Source: §A.3 · pp. 13–14, §C.1 · p. 16

C.2 Component Scores and Intervals

  • 33.5% of scenarios pass for Sonnet 4.5 (69/206), the highest observed rate; Opus 4.1 reaches 29.6%, and GPT-5 configurations range from 20.9% to 28.2%.These are recorded results on the fixed 206-scenario suite; the reported intervals do not establish pairwise model significance.Source: Figure 5 · p. 16, §C.2 · pp. 16–17

C.3 Performance Across Task Categories

Performance varies substantially across categories, models, and recorded trajectories. The analyses distinguish observed patterns from causal claims, while highlighting that clean API execution and additional tool activity do not guarantee task completion.

C.3 Performance Across Task Categories

  • Sonnet passes 88.9% of folder tasks (8/9) but 16.7% of calendar-write tasks (2/12), and its orchestration pass rate is 17.6% (3/17).Its filters and message-write rates are 80.0% and 73.3%, respectively; category contrasts show that success does not extend uniformly across write operations or coordinated workflows.Source: Table 18 · p. 17, §C.3 · p. 17
  • Pass rates are not monotonic across assigned difficulty groups: D1 is 6–12%, D2 is 28–42%, and D3 is 10–24%.The difficulty labels are assigned metadata, not calibrated thresholds for calls or domains.Source: §C.3 · p. 17, Table 19 · p. 17

C.4 Where Models Struggle

  • For the named Sonnet/Opus/GPT-5.4-R/o3 cohort, 24/206 scenarios pass for every model, 117/206 for none, and 65/206 yield differing outcomes, computed by scenario ID.This selected cohort includes o3 and is not the top four by observed pass rate.Source: §C.4 · p. 18, Table 21 · p. 19

C.5.1 Retrospective Completion by Iteration

  • By iteration five, 30/46 o3 passes (65.2%), 40/43 GPT-5 passes (93.0%), and 62/69 Sonnet passes (89.9%) are already complete.This is a retrospective analysis of runs at the original budget, not a rerun with a lower iteration cap.Source: §C.5.1 · p. 19, Table 22 · p. 19

C.5.3 Self-Correction and Recovery

  • Zero-tool failures account for 71 scenarios each for GPT-5 and GPT-5.2 (34.5% of all scenarios), versus seven for Sonnet and four for Opus.Repeated calls can reflect pagination, retrieval, mutations, or retries; call patterns alone do not establish error recognition or recovery.Source: §C.5.3 · pp. 19–20, Table 23 · p. 19

C.5.6 Score Distribution

  • A score of at least 0.5 is necessary but not sufficient to pass, because all critical assertions must also pass.The recorded OB-H307 example scores 0.50 yet fails its critical LLM rubric.Source: §C.5.6 · p. 20, Figure 9 · p. 21

C.7.1 Execution Classification

  • Sonnet has four observed failures among 1,170 calls, a 99.66% rate without observed API failure, despite a much lower scenario pass rate.Successful calls can still return incomplete data, use the wrong filter, or perform an unintended operation; these execution statistics do not recompute task grades.Source: §C.7.1 · p. 21, Table 28 · p. 21

C.8 Complexity Correlations

  • Declared domain count, difficulty, assertion count, and tool-call count describe different aspects of trajectories; their cohort comparisons are descriptive and may be confounded by task mix and tool access.The four-domain cohort contains only seven scenarios, and the analysis infers neither a fixed per-domain penalty nor a causal effect of assertion count.Source: Figure 7 · p. 18, §C.8 · p. 22

D.4 Efficiency–Accuracy Tradeoff

This appendix examines evaluation tradeoffs and limitations, including latency–accuracy, corpus scale, reproducibility, and protocol realism. It also considers how rubric design, judge-input truncation, environment limits, and differences in failures, difficulty, and iteration counts shape the interpretation of evaluation traces.

D.4 Efficiency–Accuracy Tradeoff

  • Sonnet has the highest observed accuracy and lies on the Pareto frontier, but no model is uniquely best without a specified accuracy–latency tradeoff.No other configuration is both at least as accurate and at least as fast as a frontier point, with one strict improvement; o3’s mean latency is 67.1671 s versus Sonnet’s 27.0616 s, a 2.48× ratio.Source: §D.4 · pp. 22–23

D.5 Corpus Realism vs. Scale

  • The 65-message seed corpus is deliberately small, limiting pagination pressure and needle-in-a-haystack difficulty; scaled variants with 1,000+ messages are proposed.The mock environment also lacks production network latency, rate limiting, eventual consistency, and OAuth flows.Source: §D.5 · p. 23, E Protocol-Specific Constraints · p. 23

D.6 Variance and Reproducibility

  • Only one complete result file is available per configuration, so replicate variance is unreported; static checks are deterministic, while model outputs and LLM judgments may vary.Figure 5’s Wilson intervals quantify scenario-sampling uncertainty separately. Stored scores and flags can be validated offline, but the artifacts lack an exact source-commit and tool-schema manifest.Source: §D.6 · p. 23, G Reproducibility and Availability · p. 24

E Protocol-Specific Constraints

  • The protocol evaluates English-only, single-turn, single-user, single-agent tasks and omits some tools; it does not capture iterative correction or cross-persona generalization.LLM grading may vary or miss domain-specific nuances, and the available artifacts do not establish repeated-run stability or expert agreement.Source: E Protocol-Specific Constraints · pp. 23–24

F Data Provenance Details

  • The corpus uses synthetic content inspired by public Enron data, but retains publicly documented names and identifiers; fixture discrepancies and zero-duration events may limit some evaluations.Folder metadata differ from message object counts, and several calendar events have zero duration despite rubrics expecting fractional-hour boundaries.Source: F Data Provenance Details · p. 24, §O.2 · p. 28, §O.3 · p. 28

I Batch 4 Gap-Filling Details

L.1 OB-H312: Sender Percentages

  • In the sender-percentage example, Sonnet scores 0: a substring-based static assertion fails despite an answer containing 100.0%, and the LLM judge also finds the analysis incorrect.The rubric requires calculating unread-to-total ratios per sender, applying a greater-than-50% threshold, and sorting by percentage.Source: §L.1 · p. 26

P.2 Seven Semantic Dimensions

  • The LLM assertions assess correctness, completeness, relevance, response format, and safety as semantic dimensions.These dimensions are among the seven aspects targeted by the assertions.Source: §P.2 · p. 29
  • The remaining rubric dimensions check whether requested mutations were performed and whether answers avoid fabricated data; scenario authors hand-write rubrics with exact entities, IDs, and relationships.The example cross-domain rubric specifies expected overlaps and excludes system accounts.Source: §P.2 · p. 30

P.3 Judge Input Format

  • The judge receives at most 100 characters of arguments and 200 characters of results per tool call, which can limit verification of execution details.Its input includes the query, final answer, abbreviated tool-call trace, and the rubric being evaluated.Source: §P.3 · p. 30

Q.1 Successful Trace: OB-N520 (folders, D3)

  • 5 iterations, 10 calls, and 34.886 s produced a passing OB-N520 trace, with static, LLM, and overall scores all 1.0.All five critical static checks and the critical LLM rubric passed, and returned folder IDs were used in later calls.Source: Table 36 · p. 31, §Q.1 · p. 31

Q.2 Incomplete Trace: OB-H307 (inbox-cleanup, D3)

  • 15 iterations, 29 calls, and 88.868 s ended in failure on OB-H307 despite a static score of 1.0, because its critical LLM rubric scored 0.0.The agent returned an incomplete answer without the requested final count; environment and grading limits also prevented static checks from establishing complete ordered triage.Source: Table 37 · p. 31, §Q.2 · p. 31

R Per-Difficulty Breakdown (All Models)

  • Difficulty labels alone do not explain observed D1/D2/D3 differences, because some D1 tasks request operations the adapter cannot expose.D1 therefore does not guarantee tool availability or easy grading.Source: R Per-Difficulty Breakdown (All Models) · p. 31

S Scenarios Failed by the Four-Configuration Cohort

  • 117 scenarios (56.8% of 206) were passed by none of the named Sonnet/Opus/GPT-5.4-R/o3 cohort, but their failures do not isolate intrinsic reasoning difficulty.The cohort includes unavailable operations, persona mismatches, and grading limitations, alongside potentially difficult model behaviors.Source: S Scenarios Failed by the… · p. 31

T Iteration Distribution per Model

  • Iteration counts include recorded tool-calling and terminal-answer steps, but four exception records with calls have zero stored iterations, making reconstructed counts lower bounds.A one-iteration trajectory need not indicate refusal, and a high count need not indicate an error-retry loop.Source: T Iteration Distribution per Model · p. 31

U Reproducibility Details

Reproducibility is bounded by incomplete runtime and artifact provenance, single-run evaluation, and constraints in the recorded tables and tool environment.

U Reproducibility Details

  • Recorded deployments, request settings, and timestamps do not verify the labeled configurations’ corresponding public model snapshots.The agent and judge use hosted deployments through the same bridge; one saved o3-labeled trajectory exceeds the agent subprocess limit.Source: U Reproducibility Details · pp. 31–32
  • Table 38 groups tasks by assigned difficulty, but differing tasks and tool availability prevent interpreting those differences as isolated difficulty effects.Source: Table 38 · p. 32
  • Table 39 reports overlapping scenario attributes for a selected four-configuration failure cohort, not the top four by pass rate; Table 40 gives observed iteration distributions, with last-three-column counts over all 206 scenarios.Missing stored counts are replaced by observed lower bounds from call indices; σ denotes population standard deviation.Source: Table 39 · p. 32, Table 40 · p. 32
  • The saved artifacts do not establish exact runtime versions, effective endpoint defaults, full-suite duration, or billed token usage, so no fixed hardware, runtime, or cost claim is made.Agent and judge requests omit temperature; recorded latency excludes grading calls.Source: U Reproducibility Details · p. 32
  • One recorded run per configuration leaves repeated-run variance and expert-annotation agreement unavailable.Source: U Reproducibility Details · p. 32
  • Missing source-commit and tool-schema manifests, plus potentially truncated tool responses, limit exact reconstruction despite offline reproduction of scores for all 1,648 saved results.Source: U Reproducibility Details · p. 32
  • Recomputing saved-result tables and figures is offline, but model and judge execution requires authorized credentials and access to the configured Substrate service.Source: U Reproducibility Details · p. 32

Related papers

Loading 2609.31906v1…