PaperlayerView exact revision on arXiv

Source-linked AI summary

When Agents Fail: Model or Harness--Who's to Blame?

Shao Zhang, Hangfan Zhang, Yuanyuan Shi, Hao Li, Min-Le Su, Kangcong Li, Jingping Wang, Chunjiang Mu, Chen Zhang, Yang Chen, Bo Zhang, Shuyue Hu

arXiv:2610.32483v1cs.AIcs.CLcs.SE

TL;DR

Task-centric agent benchmarks often do not separate model capability from harness effects, despite performance depending on their pairing. This paper evaluates task outcomes, valid completion, and resource use across model–harness configurations; rankings vary across models and benchmarks, supporting joint evaluation and model- and task-adapted harness design.

  • Problem

    Task-centric evaluations often summarize fixed model–harness systems by task success or reward, making it difficult to distinguish model capability from harness effects and pairing interactions.

    Source: §1 · p. 2

  • Method

    The paper evaluates crossed model–harness pairings with measures of valid completion, task performance achieved through valid completion, interaction residuals, and resource use.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    Harness rankings vary across models and benchmarks, task performance and operational validity can favor different pairings, and similar task scores can accompany distinct execution profiles.

    Source: §1 · p. 3, §7 · p. 17

  • Takeaways & Limitations

    The findings support joint assessment of task performance and operational validity, transferable harness-use training, and harness designs adapted to the model and task.

    Source: §1 · p. 3

    Each harness retains its native execution loop and bundled design choices, so comparisons concern complete model–harness systems.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.

1. Introduction

Agent performance reflects the joint behavior of a model and its harness, but task-centric evaluation often treats the harness as part of the setup. This paper proposes evaluating task achievement, valid completion, and model–harness interaction together.

1. Introduction

  • Task-centric benchmarks often treat the harness as setup, although failures and success can reflect model–harness interactions rather than model capability alone.Runs may complete validly without solving a task, or earn task credit despite abnormal termination.Source: §1 · p. 2
  • Harness difficulty depends on the model: an interface may challenge one model but suit another, and task-specific harness support can contribute to solving the task.The paper frames difficulty as an interaction cost from observation formats, action interfaces, state management, control flow, and recovery.Source: §1 · p. 2
  • The framework measures Valid Completion Rate, Model–harness Fit, and Cross-Model Fit, and examines failure patterns and resource use across crossed pairings.The study uses a common adapter while preserving native execution loops and compares four general-purpose harnesses across six benchmarks, with fixed native systems as references.Source: §1 · pp. 2–3
  • Harness rankings vary by model and benchmark, and task performance and operational validity can favor different pairings.Similar task scores can conceal distinct execution profiles; lower token use can accompany lower outcomes, while high use can include repeated failed actions.Source: §1 · p. 3
  • The findings motivate joint assessment of task performance and validity, transferable harness-use training, and harness designs adapted to the model and task.The proposed training focus includes action construction, feedback use, recovery, and stopping.Source: §1 · p. 3

2. Related Work

Prior work optimizes agent systems and reports complete-system outcomes, while emerging evaluations cross models and harnesses or add execution diagnostics. This paper complements that work by measuring task performance, operational validity, and pairing-specific effects.

Harness design and optimization

  • Agent-system optimization searches over graphs, workflows, constraints, context, harnesses, or editable agent code, complementing this paper’s focus on interaction difficulty and pairing compatibility.The cited methods report that changing the surrounding program can alter fixed-model performance.Source: §2 · p. 3
  • The related work distinguishes optimizing an agent system from evaluating how task-solving structure and compatibility vary across model–harness pairings.The paper positions its interaction analysis as complementary to agent-system optimization.Source: §2 · pp. 3–4

Harness evaluation

  • Most benchmarks report complete-system task outcomes; newer work crosses models and harnesses or adds diagnostics, resource reporting, failure profiles, and trace analyses.This paper measures task performance and operational validity across pairings and quantifies deviations from additive model and harness averages.Source: §2 · p. 4

3. Preliminaries

The paper treats an LLM agent as a model operating through a stateful harness in an external environment. It separates task performance from operational validity and compares pairings under matched conditions.

3. Preliminaries

  • Interpreting agent scores requires specifying the model, harness, and task conditions, then assessing both task performance and operational validity.The section establishes conditions for comparing model–harness pairings.Source: §3 · p. 4

3.1. Model–Harness Configurations

  • An executable agent is a model combined with a harness, while the environment holds task state and success criteria outside the harness.Model proposals are distinct from executed actions, whose results and errors feed back into later steps.Source: §3.1 · p. 4

3.2. Formal Runtime Interfaces

  • The harness is a stateful runtime with six interfaces: observation, context, control, action, update, and governance.These interfaces convert feedback, prepare model context, coordinate execution, map proposals to operations, persist runtime state, and enforce permissions and handoff.Source: §3.2 · p. 4
  • At each step, the harness builds observations and context for the model, then interprets its output, executes approved actions, and updates runtime memory.A model invocation can lead to zero or more executed actions; the environment transition describes one action.Source: §3.2 · p. 5

3.3. Operational Dimensions and Controlled Comparisons

  • Task performance measures satisfaction of the task criterion, whereas operational validity measures normal termination with an evaluator-consumable state, regardless of correctness.The two dimensions are assessed separately and jointly as task performance achieved through valid completion of the same attempt.Source: §3.3 · p. 5
  • Crossed comparisons vary model and harness independently while matching task instances and external conditions; each harness retains its native execution loop.The comparison unit is therefore the complete model–harness system, including its bundled design choices.Source: §3.3 · p. 5

4. Metric Design

The metric design separates task outcomes from valid completion, then combines them to characterize effective operation. It also measures pairing-specific VCR deviations, token demands, and uncertainty within the evaluated benchmark suite.

Task performance and operational validity

  • Valid Completion Rate measures attempts that terminate normally with an evaluator-consumable submission or final state, regardless of task correctness.Mean VCR averages these rates equally across included benchmarks for each model–harness pair.Source: §4 · p. 6

Model–harness Fit and Cross-Model Fit

  • Model–harness Fit compares a pairing’s task scores from valid attempts with the best observed harness score for that model, averaged across benchmarks.Cross-Model Fit takes the geometric mean of model-specific Fit, penalizing low Fit for a model.Source: §4 · p. 6

Joint-score interpretation and aggregation

  • Joint score counts task performance achieved through valid completion: binary tasks require both success and valid completion, while continuous tasks use valid-attempt score contributions.When VCR is positive, joint score equals VCR multiplied by the expected score conditional on valid completion.Source: §4 · p. 6

Interaction residuals

  • Interaction residuals measure panel-dependent VCR deviations from additive model and harness averages; positive values exceed the additive baseline.The averages weight the corresponding dimensions equally.Source: §4 · p. 7

Resource metrics

  • Token Load averages token consumption across scheduled attempts, including failures, and resource comparisons require complete token coverage in matched model–benchmark blocks.The uncertainty analysis uses paired task-cluster bootstrap intervals and leave-one-benchmark-out analyses to assess dependence on benchmark composition.Source: §4 · p. 7

5. Experimental Setup

The evaluation pairs models and harnesses on benchmarks with external-environment or persistent-artifact interactions, using matched tasks and controlled execution conditions. It preserves native harness interfaces and execution loops while comparing supported configurations across benchmarks.

5.1. Benchmark Suite

  • The suite targets closed-loop action in external environments or persistent artifacts, matched replay from prescribed states, and complementary harness-sensitive demands.Table 1 summarizes benchmark task counts and interaction demands.Source: §5.1 · p. 7, Table 1 · p. 7

5.2. Models and Harnesses

  • The study evaluates four general-purpose harnesses: DeepAgents, OpenCode, Pi CLI, and Codex CLI, alongside four core models and two fixed native model–harness references.The fixed references pair GPT-5.5 xhigh with Codex CLI and DeepSeek V4 Pro 0813 with DeepSeek Harness.Source: §5.2 · pp. 7–8

5.3. Evaluation Protocol

  • A common adapter matches tool permissions, resource budgets, and termination conditions while preserving native prompts, interfaces, context management, execution loops, recovery, and stopping.Original benchmark evaluators score results under unchanged task semantics and success criteria; the primary outcome is mean official task score over four repetitions.Source: §5.3 · p. 8
  • Four benchmarks use full 4×4 model–harness panels, while ClawMark and AgentSkillOS use 3×4 panels because DeepSeek V4 Flash Preview lacks required multimodal support.Cross-benchmark summaries use a common three-model, four-harness panel of 35,472 scheduled attempts in 72 configurations; fixed references and stress conditions are reported separately.Source: §5.3 · p. 8

6. Results and Implications

Across the evaluated pairings, task performance, valid completion, and resource use vary by model and benchmark; the results motivate assessing these together and adapting training and harness design to model and task.

6.1. Model–Harness Interaction

  • Harness rankings vary across models and benchmarks, and task performance and operational validity can favor different pairings.OpenCode has the highest Cross-Model Fit, while model-specific Fit profiles guide harness choice for a known model.Source: §6.1 · pp. 8–10, Figure 4 · p. 9, Figure 6 · p. 10

6.2. Joint Outcomes and Execution Diagnostics

  • Clean completion and task achievement diverge: non-clean attempts can earn task credit, while clean attempts can leave tasks unsolved.The joint-outcome view separates these execution profiles, including configurations with similar success rates.Source: §6.2 · p. 11, Figure 7 · p. 11

6.2.1. Continuous-Score Decomposition

  • On ClawMark, Kimi–DeepAgents exceeds fully clean Kimi–Codex by 46.7 Mean Score points, with nearly a third of its score from non-clean attempts.On AgentSkillOS, MiniMax–Codex’s mean within clean attempts exceeds its overall Task Score by 58.4 points, while clean attempts cover just over a third of scheduled runs.Source: §6.2.1 · p. 11, Figure 8 · p. 12

6.2.2. Execution-Stage Decomposition

  • DeepAgents loses nearly ten times as many runs before normal termination as at post-normal handoff; Codex CLI’s losses are roughly balanced between stages.Terminal-Bench 2, LOCA-Bench, and ClawMark mainly expose pre-normal failures, whereas VitaBench is near ceiling.Source: §6.2.2 · pp. 12–13, Figure 9 · p. 12

6.2.3. Case Study: Action Realization and Clean Completion

  • On ClawMark, Kimi–Codex CLI and Pi CLI achieve 100.00% and 99.25% VCR but score only 11.96 and 11.93, respectively.Strict action-realization gaps affect 6.75% of Codex CLI attempts and 40.50% of Pi CLI attempts; trajectory diagnostics complement VCR.Source: §6.2.3 · p. 13, Table 4 · p. 13

6.3. Metric Validation and Robustness

  • Joint and official scores coincide in 37 of 72 benchmark-level cells; counting only valid-completion credit reduces scores in the other 35.Both tested Fit substitutions preserve the overall ordering, while omitting any single model or benchmark leaves OpenCode first.Source: §6.3 · pp. 13–14, Table 5 · p. 14, Table 6 · p. 14

6.4. Resource Use and Repeated-Run Behavior

  • Across complete primary comparison blocks, Pi CLI leads token-use win rate, while OpenCode leads outcome-preserving win rate and Pareto-frontier share.Token Load means exceed medians in 89 of 90 fully covered primary cells; failed attempts can account for nearly all token use in a configuration.Source: §6.4 · pp. 14–15, Table 8 · p. 15, §6.4.2 · p. 16

6.4.1. Repeated-Run Outcomes and Stress Conditions

  • Run-to-run variation in task success persists alongside reliable protocol completion, while under stress VitaBench’s Cross environment is harder than every single-domain environment for every configuration.Increasing AgentSkillOS’s skill pool from 200 to 900 reduces GPT-5.5–Codex Task Score more than VCR.Source: §6.4.1 · p. 16

7. Conclusion

The conclusion argues that agent evaluation should account for model–harness interactions, because harness rankings vary by model and task and similar task scores can have different execution profiles.

7. Conclusion

  • Harness rankings depend on model and task, and similar task scores can accompany different execution profiles.The evaluation used four repetitions per task unless otherwise specified, with native prompts and internal policies retained under compatibility changes for unattended execution.Source: §7 · pp. 16–17, A. Experimental Configuration Details · p. 23

A. Experimental Configuration Details

  • The study evaluated four crossed harnesses with four models, plus two fixed native references, using specified model-serving and software configurations.DeepSeek V4 Flash Preview used its official API; MiniMax M3, Kimi K2.7, and DeepSeek V4 Pro 0813 were served locally with SGLang on eight NVIDIA H200 GPUs.Source: A. Experimental Configuration Details · p. 23, §A.1 · p. 23

A.2. Execution and Benchmark Configurations

This appendix specifies benchmark execution and scoring, then describes how task outcomes and valid completion are aggregated and compared across models and harnesses.

A.2. Execution and Benchmark Configurations

  • Runs use isolated, fresh task sandboxes and common records of submissions or states, trajectories, resource use, and termination status.The benchmarks apply shared task environments and evaluators where specified, with benchmark-specific limits and controls.Source: §A.2 · pp. 23–24
  • SWE-bench-45 uses all 45 tasks estimated to require at least an hour; Terminal-Bench 2 uses all 89 tasks, and VitaBench uses 400 English tasks across four environments.LOCA-Bench covers 75 tasks in a 256K-context setting; ClawMark covers 100 workplace tasks, while AgentSkillOS tests 30 tasks in 200-skill and 900-skill settings.Source: §A.2 · pp. 23–24

A.3. Benchmark-Specific Evaluation Metrics

  • Valid completion requires normal termination and evaluator-consumable handoff, but never task correctness; each benchmark has its own status and handoff rules.For example, SWE-bench-45 requires a patch accepted at the application stage, while AgentSkillOS requires a newly copied file under an expected output path.Source: §A.3 · p. 24

A.4. Scoring and Aggregation Details

  • Primary task outcomes average official scores over four repetitions, using binary success on four benchmarks and continuous credit on ClawMark and AgentSkillOS.Benchmarks and models receive equal weight; VitaBench weights its four environments equally, and AgentSkillOS uses the 200-skill condition as its primary outcome.Source: §A.3 · pp. 25–26, §A.4 · p. 26
  • Fit is invariant to positive score rescaling within a model–benchmark block, while interaction residuals are centered within benchmarks and token load is normalized across matched blocks.The appendix also describes outcome-only and arithmetic Fit variants and equal-weight macro-averages for execution-stage summaries.Source: §A.4 · pp. 26–27

B.1. Outcome Win Rate

  • Outcome Win Rate summarizes tie-aware pairwise outcome ordering, while model-conditioned VCR summaries report completion rates and harness comparisons across models.The outcome-rate table includes model-conditioned rates, pooled head-to-head rates, and a three-model mean; the VCR table reports six-benchmark summaries.Source: Table 9 · p. 27, §B.1 · p. 27, Table 10 · p. 27

B.3. Fixed System References

Fixed native systems and benchmark-level results show that outcome and valid completion vary across configurations; harness rankings also shift by model and benchmark.

B.3. Fixed System References

  • GPT-5.5–Codex wins most matched outcome comparisons, while DeepSeek Harness exceeds all core configurations on three benchmarks but trails MiniMax–Codex on LOCA-Bench.Native-system profiles also separate outcome from operational validity: a system can lead in task outcome without leading in valid completion.Source: §B.3 · pp. 27–28, Table 11 · p. 28

C. VCR Summaries and Sensitivity

  • Nearest-rank VCR quartiles preserve the type-7 ordering, with OpenCode leading among harnesses and Qwen among models.The appendix retains interpolated type-7 quartiles and reports observed minima for worst-configuration VCR.Source: Table 12 · p. 29, C. VCR Summaries and Sensitivity · p. 29

D.1. SWE-bench-45

  • On SWE-bench-45, Mean-of-4 leaders differ by model: Codex CLI leads for DeepSeek and Qwen, while OpenCode leads for MiniMax and Kimi.MiniMax M3 with OpenCode has the highest Best-of-4 and Mean-of-4 in its model block; its 19.4-point gap distinguishes at-least-once success from average success across four runs.Source: §D.1 · p. 29, Table 13 · p. 30
  • SWE-bench-45 VCR can diverge from task success: 25 runs that solve an issue are excluded from valid completions because they terminate abnormally.The trajectory analysis reports nearly twice as many DeepAgents RuntimeError failures for Kimi as for MiniMax, and 106 more evaluator-ready Codex CLI patches for Qwen than Kimi out of 180 attempts.Source: Table 14 · p. 30, §D.1 · p. 31

D.2. Terminal-Bench 2

  • On Terminal-Bench 2, Pi CLI has the strongest four-model Mean-of-4 macro-average, 1.3 points above OpenCode, but the leading harness varies by model.Both fixed references exceed every core configuration in outcome; GPT-5.5–Codex leads DeepSeek Harness by 13.5 points in both Best-of-4 and Mean-of-4.Source: Table 16 · p. 31, §D.2 · pp. 31–32

D.3. VitaBench

VitaBench combines strict tool permissions with substantial outcome variation despite near-ceiling valid completion; resource use also differs by model and harness.

D.3. VitaBench

  • Cross-domain interaction is the hardest VitaBench setting for every configuration; the best Cross Mean-of-4 is 20.8 points below the best single-domain setting.Across core configurations, Total Best-of-4 exceeds Total Mean-of-4 by roughly 23–32 points; DeepSeek Harness has a 25.1-point gap.Source: §D.3 · p. 32
  • For DeepSeek, Pi CLI uses less than half as many tokens per attempt as Codex CLI while trailing it by less than 0.2 percentage points in Mean-of-4.The comparison uses the same accounting rule as the cross-benchmark resource analysis.Source: §D.3 · pp. 32–33
  • Under strict tool permissions, every core VitaBench configuration has at least 98% VCR, yet Total Mean-of-4 spans 28.1 percentage points and the leading harness varies by model.Both fixed references reach 100% VCR, while DeepSeek Harness leads GPT-5.5–Codex by 7.6 points in Mean-of-4.Source: §D.3 · p. 32, Table 17 · p. 33, Table 18 · p. 34

D.4. LOCA-Bench

On LOCA-Bench, Codex CLI leads task performance within every model, while validity, model rankings, and resource use reveal distinct model–harness interactions. Harness choice accounts for more variation than model choice in this setting.

D.4. LOCA-Bench

  • Codex CLI is the best harness within every model block, leading the runner-up by 63 percentage points for MiniMax and roughly 15–20 points for each remaining model in Mean-of-4.MiniMax–Codex also exceeds GPT-5.5–Codex by 11 Mean-of-4 points and both fixed references in Best-of-4.Source: Table 19 · p. 35, §D.4 · p. 35
  • MiniMax ranks best under DeepAgents and Codex CLI, DeepSeek under OpenCode, and Kimi under Pi CLI, so model rankings depend on the execution framework.Pair-specific VCR residuals also change sign across benchmarks.Source: §D.4 · p. 35, Table 21 · p. 36
  • DeepAgents’ VCR differs by 79 percentage points between DeepSeek and Kimi, while MiniMax’s Codex CLI VCR exceeds DeepAgents’ by 65.7 points.DeepAgents’ valid-state rate equals its VCR across all four models; normal-execution failures account for most LOCA VCR losses.Source: §D.4 · p. 35, Table 20 · p. 36
  • Harness marginal-mean spreads are 2.6 times model spreads for Mean-of-4 and 3.5 times for Best-of-4; the harness main effect accounts for most aggregate variation.The reported shares describe variation among aggregated configuration means.Source: §D.4 · p. 35, Table 22 · p. 36
  • For DeepSeek, Codex CLI uses about 86% fewer tokens per attempt than Pi CLI while raising Mean-of-4 by 21.7 percentage points; tokens per success are about 92% lower.On Kimi’s 300 attempts per harness, Codex has 114 fewer runtime errors than OpenCode and no timeout-like failures.Source: §D.4 · p. 37, Table 23 · p. 37
  • Under the 256K, multi-tool, exact-state regime, harness marginal means vary more than model marginal means, while trajectory differences involve context selection, tool-result management, and stopping.Non-normal runs can retain evaluator records with zero task scores.Source: §D.4 · pp. 35, 37

D.5. ClawMark

ClawMark and AgentSkillOS show that task outcomes, operational validity, and resource use vary with both harness and model, and that skill-pool scale can change those patterns. OpenCode leads aggregate ClawMark outcomes, while AgentSkillOS harness leaders vary by model and condition.

D.5. ClawMark

  • OpenCode generally reaches high ClawMark scores with relatively compact trajectories, whereas DeepAgents’ longer execution increases exposure to timeouts and external service or transport failures.Some interrupted trajectories still receive substantial rubric credit.Source: §D.5 · p. 37
  • OpenCode has the strongest aggregate ClawMark core-panel results, while the GPT-5.5–Codex reference exceeds Kimi–OpenCode by 4.0 Mean-of-4 points and leads Best-of-4.M3 is broadly high-scoring across harnesses; K2 shows sharper harness dependence, and Qwen has broad compatibility with lower peak scores.Source: §D.5 · p. 37, Table 24 · p. 38
  • Pi CLI and Codex CLI pair near-ceiling VCR with low Kimi scores, while MiniMax and Kimi have different harnesses with positive VCR residuals.Qwen’s highest model-side VCR win rate pairs with its largest positive residual under DeepAgents.Source: Table 25 · p. 38, §D.5 · pp. 38–39
  • The native ClawMark reference leads Score Mean-of-4 while ranking below five core configurations in VCR; all 31 non-normal runs retain evaluator scores, including eight full successes.The valid-handoff rate is 100 in every configuration, so VCR is determined by normal termination.Source: §D.5 · pp. 38–39
  • For MiniMax M3, OpenCode uses about 44% fewer tokens per attempt than DeepAgents, while Kimi under Pi CLI and Codex CLI combines low Token Load with low task scores.The results illustrate why resource use should be interpreted alongside outcomes.Source: §D.5 · p. 39

D.6. AgentSkillOS

  • On AgentSkillOS, OpenCode leads MiniMax and Qwen under the primary 200-skill condition, while Pi CLI leads both under the 900-skill stress condition and leads Kimi in both.The ordering changes with skill-pool scale rather than shifting uniformly.Source: Table 26 · p. 39, §D.6 · p. 39
  • AgentSkillOS uses 1,440 scheduled attempts per skill-pool condition, with skill_200 as the primary analysis and skill_900 as a matched stress condition.The condition changes visible skill-selection burden while tasks, models, and harnesses remain matched.Source: §D.6 · p. 39
  • Raising the AgentSkillOS visible skill pool from 200 to 900 lowers OpenCode’s model-averaged VCR by 26.7 points but raises Pi CLI’s by 6.4 points.Codex CLI has negative residuals for MiniMax and positive residuals for Qwen under both conditions.Source: §D.6 · p. 39, Table 28 · p. 40

D.7. Complete Resource Results

The complete resource results cover the primary token-load configurations and a matrix spanning all benchmark, condition, model, and harness cells. LOCA-Bench token totals include consumption before abnormal termination.

D.7. Complete Resource Results

  • The primary Token Load table reports mean load for configurations with complete token accounting, including fixed system references.AgentSkillOS uses skill_200 for this primary table.Source: §D.7 · p. 40, Table 29 · p. 41
  • The complete resource matrix contains 111 benchmark–condition–model–harness cells, with outcome and exact token-coverage values for every cell.For LOCA-Bench, provider totals cover all 4,800 scheduled attempts and retain tokens consumed before abnormal termination.Source: Table 30 · p. 42, §D.7 · p. 42
  • The resource table defines Token Load in million tokens per attempt, Tokens/Success in million tokens, and coverage as the share of scheduled attempts with token accounting.Outcome is reported as official Mean-of-4 or Task Score.Source: Table 30 · p. 42
Loading 2610.32483v1…