Source-linked AI summary
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Bowei Wang, Runhui Wang, Tianyu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, Han-chung Lee
TL;DR
Existing agent benchmarks assess raw capabilities, not how much Skills improve performance. SkillsBench evaluates curated and self-generated Skills against a no-Skills baseline with deterministic verifiers. Curated Skills provide an average benefit, but effects vary, and self-generated Skills do not help on average.
Problem
Existing benchmarks measure raw agent capabilities rather than how much Skills improve task performance, leaving practitioners and researchers without empirical grounding for adoption and design decisions.
Source: §1 · p. 2Method
Results
Curated Skills provide +16.2 percentage points average benefit, while self-generated Skills provide –1.3pp average benefit and focused 2–3-module Skills outperform comprehensive documentation.
Source: §7 · p. 8Takeaways & Limitations
Skills are most helpful when success depends on concrete procedures and verifier-facing details rather than broad conceptual knowledge.
Source: §5 · p. 7Because Skills injection increases context length, observed gains could partly reflect added context; stronger length-matched baselines remain future work.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.
1. Introduction
Existing benchmarks measure agents’ raw capabilities, not how much procedural Skills improve task performance. SkillsBench addresses this gap by evaluating Skills as augmentation artifacts and previewing gains from curated Skills alongside negligible or negative benefit from self-generated Skills.
1. Introduction
- +16.2pp average improvement is reported for curated Skills, while self-generated Skills provide negligible or negative benefit.Figure 1 previews resolution rates across seven agent-model configurations and 84 tasks.Source: Figure 1 · p. 1, §1 · p. 2
- Foundation models lack domain-specific procedural knowledge, while fine-tuning is expensive and sacrifices generality.The introduction frames Skills as a possible way to supply that procedural knowledge at inference time.Source: §1 · pp. 1–2
- The benchmark is designed to examine how much Skills help, which components contribute to gains, and when Skills fail despite being present.The introduction contrasts these questions with merely asking whether task-relevant context helps.Source: §1 · p. 2
- Existing agent benchmarks assess raw model capabilities but not how much Skills improve performance on a task.The paper says this gap leaves practitioners without an informed basis for adoption and researchers without empirical grounding for Skills design principles.Source: §1 · p. 2
- SkillsBench treats Skills as first-class evaluation artifacts and evaluates curated and self-generated Skills against a no-Skills condition.Its contribution includes 84 tasks across 11 domains, deterministic verifiers, and full trajectory logging.Source: §1 · p. 2
2. SKILLSBENCH
SKILLSBENCH evaluates Skills augmentation with paired vanilla and Skills-augmented tasks, a structured Skill specification, and deterministic verification. Its construction combines community task contributions with automated checks, human review, and difficulty stratification.
2. SKILLSBENCH
- SKILLSBENCH evaluates each task with and without Skills, enabling direct measurement of Skills efficacy using deterministic verification.Tasks use containerized environments, an oracle solution, and a deterministic verifier.Source: §2 · p. 2, §2.2 · p. 3
2.1. Skills Specification
- A Skill must provide procedural guidance for a task class in a structured, portable package, with a SKILL.md file and optional resources.The operational definition distinguishes Skills from system prompts, few-shot examples, retrieval, and tool documentation, while acknowledging that the boundary is not absolute.Source: §2.1 · p. 2
2.2. Task Specification
- Each task module contains a human-readable instruction, containerized environment, reference solution, and deterministic verifier.The oracle checks that at least one correct solution path exists; programmatic assertions provide reproducible pass/fail outcomes.Source: §2.2 · p. 3
2.3. Dataset Construction
- A community-driven process gathered 322 candidate tasks from 105 contributors for curation into SKILLSBENCH.Submissions included a full task specification and author-assessed difficulty ratings.Source: §2.3 · p. 3
2.4. Contributing Principles
- Quality controls combine leakage prevention and autonomous Skill discovery with deterministic tests and automated and human review.Reviewers assess data validity, task realism, oracle and Skill quality, and anti-cheating; Skills must guide a task class rather than encode its specific solution.Source: §2.4 · pp. 3–4
2.5. Benchmark Composition
- The benchmark comprises 84 tasks across 11 domains, with difficulty stratified using estimated specialist completion time without AI assistance.Contributors provide the estimates, which are reviewed by maintainers who are experts in the relevant domain.Source: §2.5 · p. 4, Figure 3 · p. 4
3. Experimental Setup
SKILLSBENCH evaluates three commercial agent harnesses with seven model–harness configurations across three Skills conditions, using deterministic task verification and pass-rate metrics.
3. Experimental Setup
- 7,308 valid trajectories cover three commercial agents and seven frontier models under the benchmark’s evaluation setup.A trajectory is one attempt at a task under a specific Skills condition; valid attempts end in pass, fail, or timeout without infrastructure or runtime errors.Source: §3 · pp. 4–5
3.1. Agent Harnesses
- Claude Code, Codex CLI, and Gemini CLI provide the three commercial agent harnesses.Each is paired with compatible models to form seven model–harness configurations.Source: §3.1 · p. 5, §3.2 · p. 5
3.2. Models
- Seven frontier models are evaluated with temperature 0 for deterministic sampling.The set includes GPT-5.2, four Claude models, and Gemini 3 Pro and Flash.Source: §3.2 · p. 5
3.3. Skills Conditions
- Tasks are tested with no Skills, curated Skills, and self-generated Skills, where the agent creates procedural knowledge before solving.Self-generated Skills are evaluated on five of seven configurations; Gemini CLI does not support that condition.Source: §3.3 · p. 5
3.4. Evaluation Protocol
- Agents receive Skills as system-level context and work in a container until completion, timeout, or round limit; deterministic assertions determine pass or fail.The verifier produces a binary outcome.Source: §3.4 · p. 5
3.5. Metrics
- Pass rate averages binary rewards across five trials per task and then across 84 tasks; normalized gain is reported alongside absolute improvement.Normalized gain can make ceiling effects look like substantial scaffolding, so the paper reports both g and absolute improvement (∆).Source: §3.5 · p. 5
4. Results
The results examine Skills efficacy across configurations, harnesses, domains, and tasks, then test how Skills quantity, complexity, and model scale relate to performance.
4.1.1. MAIN RESULTS
- +16.2pp average improvement across seven configurations, ranging from +13.6pp to +23.3pp; Gemini CLI with Gemini 3 Flash reaches 48.7% pass rate with Skills.Claude Code with Opus 4.5 has the largest improvement, +23.3pp.Source: §4.1.1 · pp. 5–6, Table 3 · p. 6
- –1.3pp average: self-generated Skills provide negligible or negative benefit versus no Skills, unlike curated Skills.Opus 4.6 improves by +1.4pp, while Codex with GPT-5.2 declines by –5.6pp; models may produce imprecise procedures or fail to recognize when specialized knowledge is needed.Source: §4.1.1 · p. 6
4.1.2. HARNESS-SPECIFIC RELIABILITY
- Claude Code models consistently benefit and have the highest Skills utilization; Gemini CLI has the highest raw performance, while Codex CLI often neglects provided Skills.Improvements range from +13.9pp to +23.3pp for Claude Code and +13.6pp to +17.4pp for Gemini CLI; Codex CLI reaches 44.7% with Skills.Source: §4.1.2 · p. 6
4.1.3. DOMAIN-LEVEL ANALYSIS
- Healthcare (+51.9pp) and Manufacturing (+41.9pp) gain most, while Mathematics (+6.0pp) and Software Engineering (+4.5pp) gain less.The authors report larger benefits in domains requiring specialized procedural knowledge underrepresented in pretraining, and smaller benefits where pretraining coverage is strong.Source: §4.1.3 · p. 6, Table 4 · p. 6
4.1.4. TASK-LEVEL ANALYSIS
- 16 of 84 tasks have negative Skills deltas despite positive domain aggregates, including taxonomy-tree-merge (–39.3pp) and energy-ac-optimal-power-flow (–14.3pp).Largest gains include mario-coin-counting (+85.7pp, from 2.9% to 88.6%) and sales-pivot-analysis (+85.7pp); the authors suggest Skills may add conflicting guidance or unnecessary complexity on some tasks.Source: §4.1.4 · pp. 6–7
4.2.1. SKILLS QUANTITY ANALYSIS
- +18.6pp is the largest improvement for tasks provided 2–3 Skills, compared with +5.9pp for tasks provided 4+ Skills.The paper describes this as a non-monotonic relationship and suggests excess content may create cognitive overhead or conflicting guidance.Source: §4.2.1 · p. 7, Table 5 · p. 7
4.2.2. SKILLS COMPLEXITY ANALYSIS
- +18.8pp for detailed and +17.1pp for compact Skills contrast with –2.9pp for comprehensive Skills.The authors suggest agents may struggle to extract relevant information from lengthy documentation, which can also consume context budget.Source: Table 6 · p. 7, §4.2.2 · p. 7
4.2.3. MODEL SCALE EFFECTS
- 27.7%: Claude Haiku 4.5 with Skills outperforms Haiku without Skills at 11.0% by +16.7pp, while Opus 4.5 without Skills reaches 22.0%.The authors conclude Skills can partially compensate for model capacity limitations on procedural tasks.Source: §4.2.3 · p. 7
5. Discussion
Skills help most when tasks require concrete procedures or brittle-format details, but gains depend on whether the harness retrieves and follows them. The discussion recommends concise, modular, harness-aware authoring and identifies limits in causal attribution, reproducibility, and generalization.
5. Discussion
- Skills yield larger gains on specialized workflows or brittle formats, but smaller or negative effects when models have strong priors and Skills add overhead or conflict.The discussion links the pattern to whether success depends on concrete procedures and verifier-facing details, such as steps, constraints, and sanity checks.Source: §5 · p. 7
- Skills efficacy depends on harness behavior: some harnesses use retrieved Skills reliably, while others acknowledge them without invoking them.Long trajectories can also suffer format drift that reduces the influence of early-injected Skills, motivating evaluation across multiple harnesses.Source: §5 · p. 7
- Concise, stepwise guidance with a working example is often more effective than exhaustive documentation, while overly long definitions can increase context burden without improving decisions.Modular Skills appear to compose better on multi-part tasks, and guidance should match harness constraints, such as repeated format reminders for JSON-only protocols.Source: §5 · p. 7
5.1. Limitations and Future Work
- Results may not transfer directly beyond terminal-based, containerized tasks to GUI agents, multi-agent coordination, or very long-horizon workflows.The evaluation also covers a limited set of models and harnesses, whose commercial behavior and Skills integration can change over time.Source: §5.1 · pp. 7–8
- Because Skills injection increases context length, observed gains could partly reflect added context rather than procedural structure.The self-generated Skills condition suggests structure matters, but stronger length-matched baselines, such as random or irrelevant text and retrieval-only documentation, are still needed.Source: §5.1 · p. 8
- Containerization and paired comparisons mitigate, but do not eliminate, nondeterminism or training-set memorization effects.Future evaluations should use ecosystem-representative settings, including lower-quality and automatically selected Skills, and study when Skills composition helps or interferes.Source: §5.1 · p. 8
6. Related Work
SKILLSBENCH differs from benchmarks of fixed-agent task completion by measuring the efficacy of Skills augmentation. It connects procedural augmentation research with evaluation across heterogeneous models and harnesses.
Agent benchmarks
- SKILLSBENCH measures augmentation efficacy through paired evaluation, unlike benchmarks that measure how well a fixed agent completes tasks.The related-work discussion positions it alongside existing agent benchmarks, including Terminal-Bench and broader environment suites.Source: §6 · p. 8
Procedural augmentation and tool use
- Prior augmentation work includes structured reasoning, external knowledge, retrieval, and tool use, while Skills combine procedural guidance with executable resources.Despite these methods, benchmarks rarely quantify their actual impact.Source: §6 · p. 8
Skills ecosystems and evaluation methodology
- SKILLSBENCH evaluates commercial harnesses and a model-agnostic Terminal-Bench-based harness to separate model and harness effects.The related-work discussion also notes that Agent Skills and MCP specifications formalized skill packages and tool connectivity.Source: §6 · p. 8
- The paper reports absolute gains and normalized gain to support comparisons across different baselines.The discussion relates this reporting choice to broader benchmarking guidance on comparability.Source: §6 · p. 8
7. Conclusion
SKILLSBENCH evaluates Agent Skills across tasks, agent-model configurations, and three conditions. Curated Skills improve performance on average but vary in effect; self-generated Skills provide negligible or negative benefit, while focused Skills and smaller models show distinct advantages on procedural tasks.
6. Conclusion
- Curated Skills provide a +16.2 percentage points average benefit across 84 tasks, 7 agent-model configurations, and 7,308 trajectories, with high variance across domains and configurations.The evaluation compares no Skills, curated Skills, and self-generated Skills.Source: §7 · p. 8
- Self-generated Skills provide –1.3pp average benefit, and the conclusion states that effective Skills require human-curated domain expertise.This is one of four findings from the benchmark evaluation.Source: §7 · p. 8
- Across 84 tasks, 7 agent-model configurations, and 7,308 trajectories under three conditions (no Skills, curated Skills, self-generated Skills), focused Skills with 2–3 modules outperform comprehensive documentation.Source: §7 · p. 8
- Skills can partially substitute for model scale, enabling smaller models to match larger ones on procedural tasks.The conclusion characterizes Skills efficacy as context-dependent and motivates paired evaluation for agent augmentation research.Source: §7 · p. 8
Impact Statement
The impact statement frames the work as an evaluation-methodology contribution without immediate ethical concerns beyond those inherent to agent systems. The appendix analyzes the public Skills ecosystem, reporting its growth, size and structural characteristics, domain coverage, and quality scores.
Impact Statement
- The work aims to advance machine learning through improved evaluation methodology for AI agents, with no immediate ethical concerns identified beyond those inherent to agent systems generally.The impact statement says better evaluation of agent augmentation strategies may contribute to more reliable and capable AI assistants.Source: Impact Statement · p. 8
A. Skill Ecosystem Analysis
- The appendix analyzes publicly available Skills to contextualize SKILLSBENCH within the broader agent-augmentation landscape.This ecosystem analysis precedes its collection and characterization results.Source: A. Skill Ecosystem Analysis · p. 12
A.1. Data Collection
- After deduplication, the collection retained 47,150 unique Skills from 6,323 repositories; over 136 days, the cumulative total reached 84,192, with daily additions peaking at 18,904 in January 2026.Sources included public GitHub repositories, community marketplaces, and corporate partner contributions.Source: §A.1 · p. 12, Figure 5 · p. 12
A.2. Skill Characteristics
- Skill sizes follow a log-normal distribution with a median of 2.3 KB, while the largest 1% exceed 50 KB.Figure 6 shows the SKILL.md token distribution for 36,338 files, with a median of about 1.5k tokens.Source: §A.2 · p. 12, Figure 6 · p. 13
- The collection spans several domains, led by software development at 38% and data analysis at 22%; 78% of Skills follow the standard SKILL.md-plus-optional-resources structure.Most Skills contain few files, with a median of one, and the ecosystem is documentation-heavy.Source: §A.2 · p. 12
A.3. Quality Indicators
- The rubric scores clarity, specificity, and examples from 0 to 3 points each; across the ecosystem, mean quality was 6.2/12 (SD=2.8), indicating substantial room for improvement in Skill authoring practices.Source: §1 · p. 13
A.4. Implications for Benchmark Design
Ecosystem analysis shaped benchmark task selection, Skill quality, and context budgets. The authors caution that the benchmark’s high-quality Skills and matching represent an optimistic scenario for real-world use.
A.4. Implications for Benchmark Design
- Ecosystem findings guided domain selection, motivated leakage audits and authoring guidelines, led to selecting Skills from the top quality quartile (≥9/12), and informed an 8K context budget.The median ecosystem Skill size was approximately 800 tokens.Source: §A.4 · p. 13
- High-quality benchmark Skills (mean 10.1/12) contrast with the ecosystem mean (6.2/12), and real-world use also involves imperfect Skill-task matching.The authors identify these differences as a benchmark-to-ecosystem gap and call for evaluation with ecosystem-representative samples.Source: §A.4 · p. 13
B.1. Task Directory Structure
The appendix documents task construction, review, experimental execution, and task-level results. Its figures show broad but uneven gains from curated Skills, while a higher-capability pattern distinguishes easier from harder tasks.
B.1. Task Directory Structure
- Tasks use self-contained directories with metadata and resource limits, Docker setup, instructions, optional Skills, an oracle solution, and programmatic tests.A task passes only when all assertions succeed; the main evaluation awards no partial credit.Source: §B.1 · pp. 13–14, §B.3 · p. 15
B.4. PR Review Process
- Review combines structural and oracle checks with maintainer assessment of data, realism, Skills, anti-cheating, and test quality; 86 of 322 submissions were accepted.Task criteria include realistic professional workflows, deterministic verification, and meaningful Skill dependence.Source: §B.6 · p. 16, §B.4 · p. 16, §B.7 · p. 17
C.4. Skill Injection Mechanism
- Three commercial harnesses were evaluated, with Skills copied into harness-specific directories and discovered or activated by each agent at runtime.The evaluated harnesses were Claude Code, Gemini CLI, and Codex CLI; task instructions do not tell agents which Skills to use.Source: §C.2 · pp. 18–19, §C.4 · p. 19
C.6. Self-Generated Skills Condition
- Self-generated Skills require agents to write 1–5 reusable skill documents before solving; this condition was evaluated only on Claude Code and Codex configurations.Gemini CLI was excluded from this condition because it uses an explicit skill-activation interface.Source: §C.6 · pp. 19–20
C.8. Container Environment
- Trials ran in isolated Docker containers with task-dependent CPU and memory limits, deterministic sampling, an 8K-token context limit, and level-dependent round limits.Containers were deleted after each trial to prevent state leaks.Source: §C.8 · p. 20, §C.9 · p. 20, §C.10 · p. 20
C.10. Experiment Orchestration
- 7,308 valid trajectories cover 7 configurations, 84 tasks, and 3 conditions, with mean pass rates reported alongside 95% bootstrap confidence intervals.Main conditions used five runs per task-model-condition combination; a subset received 10-run validation, which left conclusions unchanged.Source: §C.10 · p. 20, §E.2 · p. 24
D. Task-Level Results
- Task-level plots show curated Skills shift many tasks from unsolved to solved, though some model-task pairs regress and task solvability tracks model capability.The plots show common easy tasks solved across models, hard tasks unsolved even with Skills, and a small number of negative deltas.Source: D. Task-Level Results · p. 20, Figure 11 · p. 21, Figure 12 · p. 22
E.2. 10-Run Validation
- In a subset of configurations and Extreme tasks, 10-run means were within 1.2pp of 5-run estimates, standard error fell about 30%, and conclusions remained unchanged.The authors conclude that five runs provide sufficient precision for the main findings.Source: §E.2 · p. 24
F. Complete Task List
The appendix catalogs the benchmark’s 86 tasks by domain and lists the 84 tasks included in evaluation. Two tasks were excluded for resource and verifier constraints.
F. Complete Task List
- 86 tasks are listed, with two excluded from evaluation: one required unavailable GPU resources and another had consistent verifier timeouts.The resulting evaluation covers 84 tasks; task difficulty was assigned by authors and validated during review.Source: F. Complete Task List · pp. 24, 26
- Tasks are grouped by domain and sorted alphabetically within each domain, with task identifiers, difficulty levels, and descriptions.The table caption identifies 86 total tasks, 84 evaluated.Source: Table 9 · p. 24, F. Complete Task List · p. 24
- The catalog spans office work, robotics, software engineering, finance, healthcare, manufacturing, mathematics, media and content, and natural science tasks.Examples include document and spreadsheet work, control and planning, code changes, data analysis, and multimedia processing.Source: F. Complete Task List · pp. 25–26
G. Comprehensive Results Summary
Curated Skills improve benchmark performance, while their cost and effects on failures vary. The analysis also identifies tasks where Skills help and tasks that remain unsolved.
G. Comprehensive Results Summary
- +16.2pp average improvement from curated Skills, ranging from +13.6 to +23.3pp across configurations.Claude Code + Opus 4.5 had the largest absolute improvement (+23.3pp); Gemini CLI + Gemini 3 Flash reached the highest with-Skills pass rate (48.7%).Source: G. Comprehensive Results Summary · p. 26, Table 10 · p. 27
H.4. Cost-Performance Tradeoff
- Skills increased input tokens by 6–13% for Codex and Gemini, but the reported performance gain outweighed the $0.03–0.22 per-trial cost increase.At standard API pricing, Gemini 3 Flash cost 47% less per task than Gemini 3 Pro ($0.57 vs. $1.06); cached pricing can reduce actual costs by 50–90%.Source: §H.2 · p. 27, §H.4 · p. 28
I.1. Infrastructure Error Summary
- Infrastructure errors included verifier timeouts, missing authentication credentials, and Gemini CLI setup timeouts, separate from agent capability failures.The verifier timeout on fix-visual-stability led to its exclusion from all experiments; Gemini CLI setup timeouts affected 14 tasks.Source: §I.1 · p. 29
I.3. Overall Failure Distribution
- Quality Below Threshold was the dominant failure mode (49.8%); across models, verification failures remained dominant at 48–53%, with timeout and execution shares varying.Failures were classified programmatically from verifier outputs into categories including execution, coherence, verification, and unknown.Source: §I.2 · pp. 29–30, §I.3 · p. 30, Table 15 · p. 30
I.5. Success Case Studies
- On three documented tasks, curated Skills raised USGS flood-risk pass rates from 2.9% to 80.0%, SEC 13F pass rates from 0% to 75.0%, and job-shop scheduling pass rates from 0% to 68.6%.For Excel pivot tables, 6 of 7 models achieved at least 80% pass rate with Skills; average improvement was +85.7 percentage points.Source: §I.5 · p. 31
I.6. How Skills Change Failure Patterns
- Skills reduced Quality Below Threshold failures by 30.8% and Incomplete Solution failures by 35.8%, while the timeout share of failures rose from 16.1% to 18.6%.The absolute timeout count nevertheless fell from 367 to 328; self-generated Skills had an 80.9% failure rate versus 78.4% without Skills.Source: §I.6 · pp. 31–32
I.7. Qualitative Failure Examples
- Qualitative examples show failures from inaccurate domain data, unfinished computational steps, formatting violations, and dependency errors in embedded spreadsheets.In the earthquake example, the agent used the correct Haversine formula but selected an incorrect plate-boundary coordinate; in the Excel example, dependent cells were not recomputed.Source: §I.7 · p. 32
I.8. Unsolved Tasks
- Sixteen of 84 tasks had 0% pass rates across models and conditions, while the ten tasks with the largest Skills gains averaged +70.4 percentage points.Unsolved tasks clustered around computationally intensive work, complex multi-step pipelines, and strict output specifications; the largest gains involved domain-specific procedural knowledge.Source: §I.8 · pp. 32–33, §I.9 · p. 33
I.10. Agent Timeout Analysis
Timeouts affected a substantial share of agent failures and clustered on particular tasks. The Gemini CLI condition also required retroactive timeout corrections and data-cleaning rules.
I.10. Agent Timeout Analysis
- 17.8% of agent failures were timeouts; 39 of 84 tasks had at least one with-Skills timeout, and three tasks timed out in all five trials across models.The timeout table reports with-Skills rates by model and includes retroactive enforcement for Gemini CLI.Source: Table 19 · p. 33, §I.10 · p. 33
- Gemini CLI recorded no AgentTimeoutErrors despite an expected 13% rate, so eight rewarded with-Skills trials were retroactively set to zero, reducing Flash and Pro by 0.7pp each.Timeout enforcement was disabled for those runs.Source: §I.10 · p. 34
I.11. Data Cleaning: Gemini CLI Retry Irregularities
- Gemini CLI irregularities included 64 launched jobs without results; missing results were scored as zero, and runs were capped at five trials per task-model.Six dropped valid trials had reward = 1.0.Source: §I.11 · p. 34