DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents

1 Introduction

• Existing benchmarks often assess static answer correctness or are tied to specific environments and protocols, limiting direct comparison and insight across modalities and execution processes.These limitations constrain support for comparable, interpretable, and reliable model selection in practice.Source: §1 · p. 1
• Agent evaluation must address workload interpretability, comparison attribution across runtimes, and trustworthy verification of execution and delivered artifacts.Final-text matching can miss unexecuted tools, timeouts, or missing artifacts, while judges may award unsupported positive credit.Source: §1 · p. 1
• Model strengths vary by workload, text and multimodal API tasks have distinct accuracy–cost frontiers, and local models are competitive in several groups but trail frontier commercial models overall.The benchmark evaluates 35 models over 7,587 model–task runs.Source: §1 · p. 2
• 233 tasks from 22 benchmarks are organized by input modality and execution form, with models evaluated in a shared OpenClaw environment under common agent-loop, tool, workspace, and logging protocols.Open-ended tasks specify the initial workspace, tools, target artifacts, scorer, and time budget.Source: §1 · pp. 1–2
• The evidence-based audit checks suspicious positive scores against execution trajectories and final artifacts, and the benchmark reports 143 unsupported positive scores removed.The audit combines deterministic post-hoc rules with a meta-judge.Source: §1 · p. 2

2.1 Agent Benchmarks and Capability Sources

• Agent benchmarks assess command-line, GUI, web, multimodal, and other capabilities, but differing runtimes, task taxonomies, and scoring protocols hinder systematic comparison.Examples include Terminal-Bench, OSWorld, MMSearch, and several OpenClaw-based benchmarks.Source: §2.1 · p. 2

2.2 LLM-as-a-Judge in Agent Evaluation

• LLM judges can infer success from plausible outputs without verifying execution evidence, while systematic characterization and mitigation of such hallucinations remain limited.Agent tasks involve long-horizon tool use, dynamic interactions, and artifact verification.Source: §2.2 · p. 2

• Figure 2 depicts DAREBench as four components: workload construction, shared execution, reliable scoring, and deployment-oriented metrics.Scoring may be automated, LLM-based, or hybrid, with evidence-based auditing; reports include token usage and reference cost.Source: Figure 2 · p. 3

3 DAREBench

• DAREBench combines workload construction, shared agent execution, reliable scoring, and deployment-oriented metrics in one evaluation protocol.It organizes 233 tasks from 22 source benchmarks into a 2 × 3 workload matrix.Source: §3 · p. 3

3.1 Problem Definition

• Each task specifies an instruction, initial workspace, available tools, artifact contract, scorer, and execution-time budget.The contract defines required output paths and validity predicates; the shared environment produces artifacts evaluated by the task-specific scorer.Source: §3.1 · p. 3

3.2 Task Construction

• Curation draws from 22 benchmarks and filters for feasibility, difficulty, and representativeness, yielding 233 tasks.Tasks lacking an explicit artifact contract or reliable scorer are removed; simple tasks are excluded, and source categories are manually sampled to avoid overrepresentation.Source: Query Selection · p. 3
• The workload taxonomy crosses text or multimodal inputs with single-step, multi-step, or tool-mediated multi-step execution to form six groups.Tasks inherit a group from their source benchmark’s dominant modality and execution form; group-level reporting reveals variation obscured by aggregate scores.Source: Workload Taxonomy · pp. 3–4

3.3 Evaluation Protocol

• Tasks use automated, LLM-based, or hybrid scoring, depending on whether their criteria can be checked deterministically, semantically, or through both modes.Automated scoring checks artifacts and verifiable intermediate states or outputs; hybrid tasks combine scoring modes with weights fixed during task construction.Source: Task Scoring · p. 4
• A meta-judge audits triggered primary-judge scores against execution evidence, setting unsupported positive scores to zero and retaining supported ones.Pilot experiments found positive judgments unsupported by trajectories or final artifacts, including credit for empty outputs or missing required artifacts.Source: Evidence-Based Audit · p. 4
• The protocol reports overall and group-level audited accuracy, average token use, and estimated per-task dollar costs.Local hosted-inference prices are third-party references rather than measured local-serving costs, so accuracy–cost frontier analyses use API reference costs.Source: Metrics and Efficiency Accounting · p. 4

4.1 Experimental Setup

• The 35-model evaluation uses a shared OpenClaw runtime; local models run in FP16 with a 32K context on one GPU, and evaluation reports accuracy, tokens, and API reference cost.API–local comparisons reflect their respective deployment profiles rather than context-matched intrinsic capability.Source: Models Evaluated · p. 5, Evaluation Metrics · p. 5

4.2 Main Results

• Token and dollar-cost extrema diverge: GPT-5.4-mini uses the fewest tokens, while MiMo-V2.5 and MiMo-V2-Omni are cheapest on text and multimodal tasks, respectively.MiMo-V2.5 uses 276k tokens overall at an overall cost of $0.005, while Claude-Opus-4.6 uses 231k tokens at $0.381.Source: Table 2 · p. 6, §4.2 · p. 6
• Leaders vary by workload, while Qwen3.6-27B leads local models at Overall 59.8 but trails frontier commercial models by about 15 percentage points.Claude-Opus-4.8 leads Text-SS, Text-MT, and MM-MT; GPT-5.5 leads Text-MS and MM-SS; Gemini-3.1-Pro leads MM-MS.Source: §4.2 · p. 6

4.3 LLM Judge Reliability

• Meta-judge auditing removes 143 unsupported positive scores; local models show deadlock loops and pseudo-tool-call credulity, while these patterns do not appear in API models.The authors frame reliability findings within the specified single-GPU local deployment profile, not as rates guaranteed under larger-context serving.Source: Table 3 · p. 6, §4.3 · p. 7

4.2 Main Results

• For text tasks, the sub-$0.02 range is near accuracy–cost saturation; compared with text, multimodal cost-to-performance gains are more pronounced, with frontier points rising from MiMo-V2-Omni ($0.003, 46.0) to Claude-Opus-4.8 ($0.123, 71.1).Source: §4.2 · p. 7

4.4 Case Study

• In the visual-search case, Qwen3.5-9B retains Avocado despite the CO2 threshold and gives a wrong answer, while Claude Opus 4.8 applies both filters and checks authoritative sources.The early chart-reading error changes the candidate set and propagates through the multi-step task.Source: §4.4 · p. 7, Figure 3 · p. 7

• DAREBench organizes 233 tasks into six workload groups and evaluates 35 API and locally deployed open-weight models using accuracy, token usage, and API reference cost.Source: §5 · p. 7

A Details of Experimental Setup

• The appendix provides reproduction details for inference configuration and the tool-and-skill suite, with model prices and judge prompts listed in cited supplementary materials.Source: A Details of Experimental Setup · p. 11

A.1 Inference Configuration

• All models use OpenClaw defaults and decode seed 42; local inference uses vLLM on an NVIDIA RTX PRO 6000 workstation without quantization or tensor parallelism.Prefix caching is enabled; the software environment is Ubuntu 24.04.4 LTS, vLLM 0.19.0, OpenClaw 2026.3.20, and CUDA 13.1.Source: §A.1 · p. 11

A.2 Tool and Skill Suite

• Models share the same default tools, web search and fetch, and nine Clawhub skills, with identical visibility and permission boundaries.Skills cover reasoning, automation, and general utilities; uniform interfaces are intended to prevent infrastructure-side bias from confounding capability comparisons.Source: §A.2 · p. 11

B.1 Model Evolution

• Qwen3.5-27B exceeds Qwen3.5-9B by 15.7 points in Overall accuracy; across Qwen3.5 and Qwen3.6, dense 27B models also outscore their 35B-A3B MoE variants.The cited comparisons are Qwen3.5: 49.5 vs. 47.2 and Qwen3.6: 59.8 vs. 56.6.Source: §B.1 · p. 11
• For Qwen3.5-27B→Qwen3.6-27B, the average multimodal gain is +11.2 versus +9.9 for text-only, while Multi-Step+Tools gains are consistently smallest.The passage describes generational gains as typically larger for multimodal than text-only tasks and says Single-Step and Multi-Step gains are relatively close.Source: §B.1 · p. 11

B.2 Error Pattern Analysis

• The failure analysis manually reviewed failed execution trajectories from 30 randomly sampled tasks stratified by task composition, then used model-assisted classification and manual auditing.Six failure patterns are presented in Table 4; Table 5 reports their distribution over zero-score trajectories.Source: §B.2 · p. 11, Table 4 · p. 12
• 28.6% of identifiable failures are PPPA, which is more frequent among local than API models (38.5% vs. 17.7% of their respective failure pools).PPPA captures long-horizon planning and contract-adherence problems, including finishing within the time limit and writing the required artifact.Source: §B.2 · p. 11
• SRU and VEB are the next most common failure modes after PPPA; SRU involves constraint violations or missed aggregations, while VEB involves bypassing visual evidence.API and local VEB counts are nearly identical (18 vs. 17).Source: §B.2 · p. 11

B.3 Local Model Token Usage and Cost

• Local models’ lower token use does not imply higher reasoning efficiency, because apparent low spend often reflects PPPA-style aborted or truncated trajectories.The section compares Qwen3.6-27B’s averages of about 62k (Text) and 81k (MM) tokens per task with many commercial API models’ over 200k under the same aggregation.Source: §B.3 · p. 12

C Details of Evidence-Based Audit

• The audit identifies five recurring judge-hallucination patterns, including rubric leakage, deadlock loops, pseudo-tool-call credulity, output instability, and timeout-with-credit.A deterministic high-recall pre-filter routes potential cases to an independent meta-judge for second-line adjudication.Source: C Details of Evidence-Based Audit · p. 12
• The pre-filter recalled 182 potential hallucination cases among 3,340 judge or hybrid calls, and the second-line review confirmed 143 as judging hallucinations.The audit supplements the Evidence-Based Audit described in the main paper.Source: C Details of Evidence-Based Audit · pp. 12–13
• 0.049 is the MAE between meta-judge-corrected and human scores, approximately half the primary judge’s 0.098 MAE.Experts independently scored a stratified sample of 100 trajectories drawn from 3,340 judge or hybrid calls.Source: C Details of Evidence-Based Audit · p. 13

D Case Study

• The supplementary case studies contrast successful API and failed local trajectories and include multimodal failures involving chart alignment, chemistry bookkeeping, and plate-map grounding.The examples also illustrate cross-step state loss in a tool-mediated text task and visual-symbolic grounding failure in a compact multimodal task.Source: D Case Study · p. 13
• In one case, the judge awarded full credit for a correct answer embedded in tool-like text despite no actual workspace file write.Figure 5 illustrates pseudo-tool-call credulity.Source: Figure 5 · p. 14
• In another case, the judge marked an incorrect “No anomalies are detected” response correct after mistaking the rubric’s reference answer for the agent’s response.Figure 6 illustrates rubric leakage.Source: Figure 6 · p. 14

D.1 Case 1 — A Multi-Hop Academic Author Retrieval Workflow

• The task required finding the most-cited author in the first article of the 65th Journal of Artificial Intelligence Research edition in 2019.The successful trajectory compared citation counts for the target article’s authors before writing the answer.Source: §D.1 · p. 15
• Gemini 3.1 Pro correctly identified Joelle Pineau as the most cited author after confirming the first article and extracting its five authors.It located the journal’s 65th volume as 2019, retrieved the table of contents, and confirmed the target article before querying author citations.Source: §D.1 · p. 15
• Qwen3-VL-32B returned an incorrect answer after mistaking a different article’s authors for the target article’s authors and continuing with that mistaken set.Although it later retrieved the correct first-article information, the earlier error was not overwritten.Source: §D.1 · p. 15

D.2 Case 2 — Academic Diagram Understanding

• Gemini 3.1 Pro answered the diagram question correctly, while Qwen3-VL-32B mistook the “fi” marker for the requested curve identity.The task asks which line is furthest from its fi value from the W-H axis; the correct reasoning compares the curve distances using the legend’s lambda values.Source: §D.2 · pp. 15–16

D.3 Case 3 — Shared Chart-Grounding Failure

• For the 5-way full-shot chart question comparing accuracy decline from Session 1 to Session 9, the reference answer is Joint-CNN; Ft-CNN was a plausible but incorrect selection.Source: §D.3 · p. 16

D.4 Case 4 — Image-Grounded Chemistry Formula Inference

• In the chemistry item, the reference molecular formula is C23H22NO4+, while the models’ submitted formulas had incorrect atom counts or did not match the reference.MiMo-V2.5 submitted C29H32NO7+ and Qwen3-VL-32B submitted C31H34NO8.Source: §D.4 · p. 16

D.5 Case 5 — Spatial Plate-Boundary Reasoning

• 0.0: Claude Opus 4.6 and Qwen3.6-35B-A3B both answer incorrectly; the reference is D, South Kesh Plate and Eurybian Plate.The item asks which of nine named boundaries might support the planet’s longest range of tallest mountains, assuming Earth-like plate geology.Source: §D.5 · pp. 16–17
• Claude applies a strong-convergence mountain-building criterion but selects a boundary other than the reference.The trace describes tallest ranges forming along strong convergent or collision boundaries, yet its selected option is not D.Source: §D.5 · p. 17

E OpenRouter Pricing

• Reference API costs use input, output, cache-read, and cache-write token rates; when cache rates are unavailable, those tokens use the input price.OpenRouter listing prices are used except for three models priced at their official provider rates.Source: E OpenRouter Pricing · p. 17

F.1 Benchmark Composition

• The 233 tasks map to six groups: Text-SS 20, Text-MS 87, Text-MT 55, MM-SS 20, MM-MS 26, and MM-MT 25.The groups cross text or multimodal input with single-step, multi-step, or tool-mediated multi-step execution.Source: §F.1 · p. 17

F.2 Benchmark Scoring

• Automated scoring handles rule-verifiable contracts, LLM judging handles open-ended or semantic criteria, and hybrid scoring combines both with fixed task weights.Judge and hybrid scores also pass through the evidence-based audit.Source: §F.2 · p. 17

F.3 Source Benchmarks

• The 22-source catalog spans reasoning, long-context and search tasks, instruction following, specialist domains, coding, desktop control, visual QA, and multimodal search.Its examples include multi-hop financial QA, terminal workflows, desktop interaction, expert multimodal QA, and cross-modal search.Source: §F.3 · pp. 17, 19, 21
• Figure 7 partitions the 233-task suite by modality and execution complexity, with segment sizes proportional to task counts and an outer ring for 22 source benchmarks.The inner ring shows the 2 × 3 workload taxonomy; the outer ring identifies contributing benchmarks.Source: Figure 7 · p. 19

F.2 Benchmark Scoring

• 125 tasks (53.6%) use automated scoring, 73 (31.3%) LLM judging, and 35 (15.0%) hybrid scoring.The distribution varies by workload group; MM-SS is exclusively judge-scored, MM-MT exclusively hybrid, and Text-MS and Text-MT predominantly automated.Source: §F.2 · p. 17, Table 8 · p. 19

G Prompts

• Appendix G documents the LLM-as-a-Judge grading prompt and meta-judge auditing prompt; the experiments use OpenClaw’s default system prompt, which is not reproduced.Figures 8 and 9 show the respective prompt templates.Source: G Prompts · p. 21, Figure 8 · p. 22, Figure 9 · p. 23