AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

1 Introduction

• Comparing agents requires controlling session reuse, preserving harness-native behavior, and linking scores to the actions, failures, and resource use of the same attempt.Uncontrolled reuse can expose earlier solutions and make comparisons depend on task order.Source: §1 · p. 2
• AlphaDiana links controlled agent comparisons and matched component interventions to outcomes, traces, and resource use.Interventions vary tools, skills, or memory while fixing the model, tasks, scoring, and decoding.Source: §1 · p. 2
• Its modular interfaces control file resets and memory retention separately, preserve each harness’s execution loop, and link scores and execution status to traces and resource use.Reports include failed and unscored results and use the latest record per requested task evaluation, so retries do not count as extra evaluations.Source: §1 · p. 2
• Harnesses can add solutions while lowering accuracy: on Qwen/GPQA-Diamond, each harness solves 8–13 questions missed by Direct but loses 18–36 Direct solves.Across four systems, 175/198 questions are solved at least once, versus 159/198 for Direct alone.Source: §1 · p. 3

2 AlphaDiana: System Design and Interfaces

• Configurations select tasks, model–harness pairs, environments, scorers, analyses, resource limits, evaluation counts, parallelism, and memory scope; the runner schedules and scores each evaluation as a sample.Retries retain the sample identifier, linking available outputs, errors, and termination details to the same evaluation.Source: §2 · pp. 3–4
• Session management assigns attempts to isolated or shared sessions while controlling file resets and memory retention separately.Shared-memory attempts run sequentially and are not automatically retried because a failure may already have changed memory inherited by later attempts.Source: §2 · p. 4
• Adapters translate tasks and normalize outputs while preserving harness-specific model calls, permissions, feedback, context updates, and tool-action handling.Harness runtimes can be local, containerized, or remote; inference services are independently local or remote.Source: §2 · p. 4
• Normalized traces link actions and feedback to source records, while outcome records separate scorer judgments from execution status.The analyzer compares token use and reasoning/tool-step patterns; entropy measures concentration among recorded top-K next-token candidates.Source: §2 · p. 4
• The modular interfaces support macro comparisons and matched interventions on tools, skills, and memory, with execution and scoring artifacts analyzed for accuracy, behavior, and resource use.New summaries reuse these records; component changes require matched executions.Source: §2 · p. 4

Benchmark setup

• The study pairs three models with three harnesses on five answer-based benchmarks and tests Qwen and Gemma on two interactive benchmarks, using each harness’s default configuration.The analysis compares completed agents and examines error labels, functional steps, token entropy, and resource use; AIME 26 uses Avg@4 and Pass@4, while other answer-based benchmarks use Avg@1.Source: §3 · p. 4

Error patterns

• On HLE-Verifiable, OpenCode and ZeroClaw have similar overall repetition rates (36.8% and 39.4%), although OpenCode’s rate is lower within every model.Budget-exhaustion labels occur in 13.0% of OpenCode’s annotated incorrect HLE attempts and none of ZeroClaw’s, so repetition alone does not measure resource consumption.Source: §3 · p. 5, Figure 6 · p. 7

Agent performance

• All three harnesses lower Qwen’s scores on five answer-based benchmarks, while ZeroClaw improves Gemma on all five; on Qwen/GPQA-Diamond, harnesses solve 8–13 questions Direct misses but lose 18–36 Direct solves.Together, the four systems solve 175/198 questions, versus Direct’s 159/198; neither tested harness exceeds the task-native reference on either interactive benchmark.Source: §3 · p. 5, Figure 5 · p. 6

Functional-step patterns

• For incorrect Gemma/ZeroClaw HLE-Verifiable trajectories, the 61.0% pooled Tool Use share comes from one of 75 trajectories; averaging trajectory shares gives 1.3%.OpenClaw has consecutive Verification steps in both outcome groups, but distinguishing correction from repeated failed checks requires inspecting what each check found and how the model used it.Source: §3 · p. 6, Figure 7 · p. 7

Token-entropy patterns

• Lower token entropy can accompany incorrect solutions: it often falls after tool feedback for both correct and incorrect trajectories, and long incorrect Direct and OpenCode trajectories can have low mean entropy.The analysis uses renormalized top-K next-token probabilities for Qwen and Gemma on AIME 26, GPQA-Diamond, and HLE-Verifiable.Source: §3 · pp. 6–7, Figure 8 · p. 8

Token use and wall-clock time

• On Qwen/AIME 26, OpenCode uses 24.6% fewer output tokens than Direct but takes 49.6% longer, illustrating that token and time measures can rank systems differently.Both measures include unsuccessful attempts; wall-clock time includes model serving, tool execution, and scheduling.Source: §3 · pp. 7–8, Figure 9 · p. 8

4 Micro View: How Harness Capabilities Contribute to Reasoning

• Tool removal lowered ZeroClaw accuracy in three of four model–task pairs but raised OpenCode accuracy in three.On AIME 26, Kimi/OpenCode had 116 correct attempts versus 104 by default, yet solved 29 problems at least once versus the default’s 30.Source: §4 · p. 9, Figure 10 · p. 9
• Only 2 of 16 skill-loading comparisons gained average accuracy, both with Qwen; Kimi/ZeroClaw read Math on every GPQA attempt while accuracy fell.Reading instructions differed by harness: OpenCode matched instructions across libraries, while ZeroClaw required Math reading but made General optional.Source: §4 · p. 9, Figure 11 · p. 9, Figure 12 · p. 10
• Across all three persistence scopes, memory helped OpenCode and hurt OpenClaw and ZeroClaw, while all nine configurations remained below Direct.Cross-Task accuracy averages tasks of differing difficulty, and later attempts can reuse prior solutions rather than being independent.Source: §4 · pp. 9–10, Figure 13 · p. 10, Figure 14 · p. 10
• On HLE-Verifiable, OpenCode used 0.6% fewer output tokens per attempt than Direct but 16.5% more per correct answer.Output tokens per correct answer divide output tokens per attempt by accuracy as a fraction.Source: §4 · pp. 10–11, Figure 15 · p. 11
• On HLE-Verifiable, unsuccessful attempts consumed 89.8% of OpenClaw output; its highest-token 10% consumed 62.2% without a correct answer.The unsuccessful-attempt and highest-token groups overlap, so their output shares cannot be added.Source: §4 · p. 11, Figure 16 · p. 11

5 Conclusion

• AlphaDiana links harness-native execution and controlled session reuse to outcomes, traces, and resource records.Matched interventions and saved traces let researchers test changes and inspect individual failures.Source: §5 · p. 11
• Harnesses can solve questions missed by the model alone yet reduce accuracy, so component changes should be judged by correctness and total output per correct answer.Fewer output tokens per attempt can mean more output tokens per correct answer.Source: §5 · p. 11

Appendix

• Appendix A documents interfaces, B experimental settings and measurement coverage, C analysis and supplementary macro results, D micro-study protocols, and E related work.Source: Appendix · p. 24

A.1 Overview

• Separating benchmarks, agents, environments, scorers, and runner orchestration lets the same benchmark and scorer be reused across model-harness systems for more attributable comparisons.Source: §A.1 · p. 25

A.2 Dataset APIs

• The benchmark API standardizes task inputs and references while keeping harness prompts, tool policies, execution, and scoring logic outside benchmark definitions.Source: §A.2 · p. 28

A.3 Agent Module

• A uniform agent interface supports direct and harnessed modes, recording predictions, trajectories, termination reasons, and execution metadata for outcome and process analysis.Source: §A.3 · pp. 29–30

A.3.1 Matched Harness Trace

• In one matched Qwen3.5-27B HLE example, all three harnesses used local computation, obtained minb(t) ≈1.0346, and selected E through different native execution paths.Source: Table 2 · p. 31, §A.3.1 · p. 31

A.3.2 Measured Extension Footprints

• Extension examples reuse the runner and result-storage code, while integration work varies by domain and whether an existing environment or verifier can be reused.The examples cover a benchmark adapter for MMMU-Pro, environment and verifier support for Terminal-Bench 2, and an agent adapter for Pi on AIME 2024.Source: §A.3.2 · pp. 31–32, Table 3 · p. 32

A.4 Environment Module

• The optional sandbox environment executes agent actions behind a portability boundary, with explicit reset and controlled reuse to limit hidden state and improve reproducibility.AlphaDiana’s default ROCK-based backend isolates actions from the host and records resource profiles and image provenance.Source: §A.4 · pp. 32–33, Figure 19 · p. 32

A.5 Scorer Module

• Outcome scoring judges task correctness independently of harness self-judgment, while separate post-execution analyzers measure trajectories, behavior, and resource use.The scorer returns a ScoreResult with a binary correctness label and scalar score; built-in approaches include exact-match, math-aware, and LLM-as-judge scoring.Source: §A.5 · p. 34

A.6 Runner and Execution Records

• The runner saves attempt-level records that support accuracy, completion, reliability, resource-use, and multi-sample metrics without relying on transient in-memory state.Records retain task identity, prediction, score, trajectory, metadata, resource metrics, and termination information; retries do not count as additional samples, and failures and unscored outputs remain available.Source: §A.6 · pp. 36–37

A.7 Run Configuration and Inspection

• The dashboard example links explicit run configuration to result inspection, while saved task and sample identifiers connect outcomes to traces for offline analysis without rerunning inference.The illustrated dashboard reports 30/30 completed tasks, zero failed runs, 97% Pass@4, and 93% Avg@4 for its example run.Source: §A.7 · p. 38, Figure 22 · p. 38

B Experimental Setup and Reproducibility

• The appendix specifies shared experimental settings, resource limits, and measurement coverage for Sections 3 and 4, with analysis methods and supplementary macro results in Appendix C and intervention configurations in Appendix D.Source: B Experimental Setup and Reproducibility · p. 39

B.1 Evaluation Setup

• Macro runs pair three models with three native harnesses across five answer-based benchmarks and test two models on two interactive benchmarks.Direct provides the answer-based reference; interactive evaluations use task-native references.Source: §B.1 · p. 39
• Table 4 lists each benchmark’s task inputs, outcome scorer, and reported metrics.Its rows follow the order of benchmarks in Figure 5.Source: Table 4 · p. 39

B.2 Resource Measurement and Observed Task Sets

• Recorded output tokens exclude inputs, while output-token and wall-clock measures include work on unsuccessful attempts.Token capture is incomplete for several model-task blocks, so token means include only attempts with recorded counts.Source: §B.2 · p. 39

B.3 Reproduction Resources and Configuration Scope

• Reproduction requires retaining each model-task block’s endpoint, harness version, task list, scorer, decoding, sample identifiers, and resource limits.Resource summaries must also follow the stated coverage and denominator conventions.Source: §B.3 · p. 41

C Macro Study: Analysis Methods and Supplementary Results

• Macro-study analyses cover benchmark performance, error patterns, functional-step patterns, token entropy, and token use and wall-clock time.Appendix B gives evaluation and resource-coverage conventions; Appendix D covers component interventions.Source: C Macro Study: Analysis Methods and… · p. 41

C.1 Benchmark Performance and Task Coverage

• Benchmark scores summarize outcomes, while paired comparisons identify task-status changes and an extraction ablation tests post-execution answer processing.The paired GPQA-Diamond accounting makes task additions and losses explicit.Source: §C.1 · p. 41, §C.1.1 · p. 41

C.1.1 Paired Coverage and Token Use on GPQA-Diamond

• 175/198 tasks are solved by the four-system union (88.4%), versus 159/198 (80.3%) by Direct; each harness loses more Direct-correct tasks than it adds.Paired accounting separates added and lost tasks from output-token use on wrong versus correct trajectories.Source: Table 6 · p. 41, §C.1.1 · p. 41

C.1.2 Paired Outcomes on Interactive Tasks

• On Qwen/Terminal-Bench 2, terminus-2 solves 35/89 tasks, OpenClaw 26/89, and ZeroClaw and OpenCode 13/89 each.Across 88 paired tasks, OpenClaw gains seven solutions and loses 16 against terminus-2; the Gemma four-system union solves 25/89. Exact McNemar tests (n = 86) against terminus-2 give p = 0.021 for OpenCode, 0.001 for ZeroClaw, and 0.115 for OpenClaw.Source: §C.1.2 · p. 42

C.1.3 Answer-Extraction Ablation

• Table 7 reports percentage-point score changes after removing final-line and final-integer fallbacks, retaining only boxed or explicitly marked answers.Model outputs, references, and scorers remain unchanged; zero indicates identical aggregate scores under strict and reported extraction.Source: Table 7 · p. 42, §C.1.3 · p. 42

• The analysis labels 2,857 incorrect attempts across five families, and its rates describe this sample rather than full-archive prevalence.The families are Specification and State Design, Reasoning and Control, Tool and Evidence Interaction, Verification and Finalization, and Resource and Operational Reliability.Source: §C.2 · p. 42
• On HLE-Verifiable, OpenCode’s repetition rate is lower than ZeroClaw’s within each model: 19.0% vs 33.3% for Gemma, 16.2% vs 24.4% for Kimi, and 66.0% vs 76.6% for Qwen.Combined rates are 36.8% for OpenCode and 39.4% for ZeroClaw; differing model proportions narrow the combined gap.Source: §C.2 · p. 43
• Sensitivity checks changed overall code rates by less than 0.54 percentage points, and the ordering of the three leading labels remained unchanged under two alternate analyses.Excluding image-supplemented attempts or equally weighting all 48 combinations preserved that ordering.Source: §C.2 · p. 43
• Codes are overlapping observable indicators requiring stated evidence; an incorrect answer alone does not establish a process error, and missing evidence does not prove absence.Each attempt counts at most once per code, including errors later recovered; percentages need not sum to 100%.Source: §C.2 · p. 43
• The taxonomy distinguishes reasoning drift and repetitive cycles from premature commitment, answer-format failures, budget exhaustion, and runtime disruption using behavior- or status-specific evidence.Optional reasoning subtypes are retained only when evidence supports them; overlapping incidents can receive multiple labels when each is supported.Source: §C.2 · pp. 44–45

C.2.1 Human Alignment of Error Labels

• Screening retained 42 of 60 selected failure trajectories, covering 40 problems; nine unresolved outcomes and other task-validity or outcome issues were excluded.The primary comparison used 714 paired decisions per rater pair on the same trajectory and code.Source: §C.2.1 · p. 45
• Mean positive F1 was 0.608 for LLM–human comparisons, versus 0.548 between humans; aggregate agreement was comparable to inter-annotator agreement.Reference-specific agreement was 84.5% with H1 and 91.5% with H2.Source: §C.2.1 · p. 46, Table 11 · p. 46, Table 12 · p. 46

C.3 Functional-Step Annotation and Aggregation

• The functional-step analysis covers 2,751 scored trajectories across two models, three benchmarks, four systems, and two outcomes; its pooled summaries give longer trajectories more weight.Adjacent-label transitions are counted within HLE trajectories, including self-transitions but not cross-trajectory edges.Source: §C.3 · pp. 46–47, Table 13 · p. 46
• Functional steps are saved assistant-text segments or logged tool calls; each receives one context-supported label, and labels are categories rather than a prescribed sequence.Tool-call targets always receive Tool Use, including denied calls; functional labels differ from multi-label error judgments.Source: §C.3 · pp. 46–47, Table 14 · p. 47
• Kimi/ZeroClaw’s incorrect HLE Verification share is 3.4% with equal-trajectory weighting, versus 26.7% when pooling steps.One trajectory contributes 211 of 217 Verification steps, showing how pooled counts can concentrate checking from a single long trajectory.Source: §C.3 · p. 47

C.3.1 Human Alignment of Functional-Step Labels

• The classifier matched human majorities on 50/55 machine-hidden spans (90.9%; κ = 0.868), while machine-visible majority-reference agreement was 73.2%.In the machine-visible phase, mean LLM–human agreement was 67.2%, versus 67.7% between humans; the two phases used separate fixed-span sets.Source: §C.3.1 · p. 48

C.4 Token-Entropy Patterns

• Entropy patterns varied by harness: OpenClaw entropy fell with tool depth, while OpenCode wrong-trajectory entropy peaked near four calls; low entropy could also accompany incorrect answers.Under OpenClaw, correct and wrong curves remained close. In OpenCode, the correct curve moved above the wrong curve at 6+ calls; across output length, wrong trajectories had low entropy under Direct and OpenCode but higher entropy than correct ones under ZeroClaw.Source: §C.4.1 · p. 48, §C.4.2 · pp. 48–49, Figure 24 · p. 49

C.5 Token Use and Wall-Clock Time Sensitivity

• A smaller time cap retained at least 99% of original accuracy in 35/36 cells; in eight blocks, system rankings stayed unchanged across qualifying caps.The exception was Qwen3.5-27B/GPQA-Diamond/OpenCode, which remained right-censored at 9,300 seconds.Source: §C.5 · p. 50

D Component Intervention Protocols

• Tool removal and skill loading were matched interventions that retained the model, task, scorer, decoding, budget, and agent loop; skill loading added a library pointer while leaving native tools available.OpenCode used the same loading-instruction wording for both libraries, while ZeroClaw required Math loading and made General optional.Source: Table 17 · p. 51, §D.2 · p. 51, §D.3 · p. 51

D.4 Memory Mechanisms and Persistence

• Memory conditions varied retention across cross-rollout, cross-task, and frozen cross-dataset settings, retaining only designated memory while resetting raw histories and workspace files.Default Harness clears memory and does not explicitly elicit its use, so comparison with Memory-Elicited combines instruction and persistence effects; cross-harness comparisons cannot isolate memory quality, retrieval, prompt length, or control logic.Source: §D.4 · pp. 51–52

D.5 Output-Token Use Across Full Harness Configurations

• Output-token comparisons cover full harness configurations, not isolated components, and tokens per correct answer include tokens spent on unsuccessful attempts.High-token and unsuccessful-attempt shares overlap, so the comparisons describe tokens used to obtain correct answers rather than causal component overhead.Source: §D.5 · p. 52

E.1 Reasoning Foundation Models; E.2 Agentic Reasoning Systems and Open Agents

• Model-only evaluations measure standalone reasoning, while complete-agent studies typically do not isolate harness effects; AlphaDiana addresses this with matched system comparisons and component tests.The related-work discussion also identifies a need for a general substrate supporting shared tasks, verifiers, and budgets across heterogeneous systems.Source: §E.1 · p. 53, §E.2 · pp. 53, 55

E.3 Reasoning and Interactive Evaluation Benchmarks

• Model-centric benchmarks assess reasoning under fixed inference protocols, whereas interactive benchmarks evaluate models embedded in harnesses that can act, observe, and interact with environments.Model-centric benchmarks alone do not distinguish model capability from harness capability when tools, memory, execution support, and multi-step interaction mediate reasoning.Source: §E.3 · pp. 55–56
• Interactive benchmarks span web, computer-use, workplace, coding, research, and multi-agent tasks, including WebArena [129], OSWorld [101], SWE-bench, and MultiAgentBench [131].Newer workflow-oriented benchmarks also emphasize structured tool use, conversational state changes, and policy-constrained interaction [109] [98] [5] [81].Source: §E.3 · p. 56
• Interactive benchmarks measure end-to-end behavior, but outcomes also depend on harness tools, control flow, observation formatting, memory and state handling, and runtime policies.This makes them valuable for full-system measurement while motivating frameworks that separate model effects from harness effects.Source: §E.3 · p. 56
• Model-centric benchmarks test isolated inference, whereas interactive benchmarks test model-harness systems; verifiable reasoning links correctness signals with system-level effects.The paper’s empirical design uses largely verifiable reasoning settings to retain reliable correctness signals while surfacing effects of embedding models in harnesses.Source: §E.3 · pp. 56–57

E.4 Evaluation Protocols, Reliability, and Human–Agent Interaction

• Evaluation work extends beyond accuracy to stepwise tool use, repeated-trial reliability, trajectory judging, robustness, recoverability, efficiency, and protocol sensitivity.Examples include T-Eval [11], τ-bench [109], and AgentRewardBench [61]; related work also studies when human collaboration is preferable and how human feedback can improve test-time behavior [68] [32].Source: §E.4 · p. 57
• Existing protocols strengthen trajectory-aware measurement and reliability analysis, but generally do not isolate performance attributable to the harness.They improve measurement of end-to-end agent behavior without resolving attribution when a model runs inside a complex runtime.Source: §E.4 · p. 57

• Comparisons of model–harness systems require explicit session reuse, memory retention, and resource-limit policies, with task outcomes linked to execution records.Source: §E.5 · p. 57
• AlphaDiana compares complete agents and intervenes on tool, skill, and memory configurations while preserving harness-native execution and separately controlling file reuse and memory retention.Source: §E.5 · p. 57
• The interfaces support comparisons such as Qwen3.5-27B with OpenClaw on HLE-Verifiable under explicit task, scoring, and resource policies.Source: §E.5 · p. 57