PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills

Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, Seong Hee Lee

arXiv:2608.20614v1cs.AI

TL;DR

Existing skill gates provide structural, documentation, and security evidence but do not measure whether a capability package improves live-agent task completion under controlled deployment conditions. ACES addresses this gap by running paired with-skill and baseline trials with shared assets and grading, finding positive average Skill Lift across a broad production corpus while showing important scope and model-update boundaries.

  • Problem

    Current skill evaluation scans structure, documentation, scripts, and security but does not reveal how an agent discovers, invokes, or uses a skill at runtime.

  • Method

    ACES runs paired with-skill and baseline trials under the same task, agent, model, assets, and grading policy, using ATIF-compatible trajectories and shared evaluation contracts.

  • Results

    Mean composite Skill Lift is 0.2134 across 947 scored paired cases from 58 of 64 production skills and four primary harnesses.

  • Takeaways & Limitations

    Live evaluation adds evidence about discovery, routing, workflow following, and tool use that scan-only gates cannot observe.

  • Takeaways & Limitations

    The results come from a corpus skewed toward infrastructure-oriented skills and four primary harnesses, so broader replication is needed for generalizability.

Abstract

from arXiv · show

Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $ρ= 0.14$). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8\% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.

1 Introduction

ACES argues that static scans cannot establish whether skills help live agents complete tasks, so it evaluates skills as executable artifacts through paired runtime trials and shared trajectory grading. Its study finds scan methods only weakly agree, while live evaluation measures positive marginal skill value across production cases.

  • Motivation: Skills package procedural know-how, scripts, and references that agents progressively disclose at inference time.This extension mechanism lets many skills coexist without bloating the agent’s context.
  • Current evaluation practice: Static tooling combines structural checks, LLM-as-Judge documentation rubrics, linters, and security scanners.These methods inspect specifications, clarity, scripts, and dangerous patterns, but remain artifact-level evaluations.
  • The runtime gap: Runtime failures include missed discovery, incorrect script invocation, output misreporting, workspace collisions, and model-update regressions that scans cannot observe.The deployment question is therefore behavioral: how an agent uses the skill under live conditions.
  • Empirical motivation: 94.5% of skills pass the default structural gate, while deterministic and LLM-judge scores correlate at Spearman ρ= 0.14.The corpus contains 145 real skills from internal enterprise repositories and public catalogs; scan methods provide complementary rather than convergent quality signals.
  • ACES: ACES evaluates skills as executable agent artifacts using an evaluation-asset contract, paired runtime tasks, and an ATIF-based trajectory contract.With-skill and baseline conditions differ in target-skill availability while shared grading compares behavior across harnesses.
  • Results: Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134.The reported 95% paired-case CI is [0.1967, 0.2301], and mean outcome-only lift is 0.1799.

2 Evaluating the Skill Artifact

The paper evaluates skills through complementary static checks and LLM judging, then shows that these artifact-level methods leave runtime behavior unobserved. Across 145 skills, permissive structural gates and weak agreement between scan methods motivate live evaluation.

  • 2.1 Static Structural Quality: ≈50 deterministic rules score skills across four weighted dimensions, with a default gate accepting scores at 70 or above.The checks are reproducible, run in milliseconds, and deduct severity-weighted findings from dimension scores.
  • 2.2 LLM-as-Judge Clarity: What Static Cannot See: LLM-as-Judge adds subjective rubric scoring for documentation clarity, examples, scope, trigger phrasing, and structural coherence.The judge reads SKILL.md, optional scripts, and structural results, returning per-criterion scores from 0–10.
  • 2.3 Linting and Security Scanning: Script linting and security scanning inspect executable content and detect issues such as syntax problems, prompt injection, leaked secrets, destructive commands, and supply-chain risks.These checks complement structural and rubric evaluation but remain scans of static artifacts rather than live skill use.
  • 2.4 What Doc-Scanning Tells Us—and What It Misses: 94.5% of 145 skills pass the 70-point structural gate, but only 48.9% reach 80 points.Structural scores range from 61 to 98, with mean 79.2, median 79.8, and σ=4.9.
  • 2.4 What Doc-Scanning Tells Us—and What It Misses: 99.3% omit tool declarations, 97.9% omit a Limitations section, 97.2% omit author information, and 91.7% omit tags.These frontmatter violations show that the declared contract is aspirational under the default gate, motivating configurable strict-mode gating.
  • 2.4 What Doc-Scanning Tells Us—and What It Misses: Structural and LLM-judge scores correlate weakly at Spearman ρ=0.14 and Pearson r=0.08 on the same 145 skills.The two methods pass 94.5% and 86.2% at threshold 70, respectively, yet neither observes discovery, script invocation, output interpretation, interference, or task success.
  • 2.4 What Doc-Scanning Tells Us—and What It Misses: Prior tools cover only one or two of five capabilities: artifact scoring, marginal-value measurement, multi-harness evaluation, CI-native deployment, and human-in-the-loop refinement.The capability gap motivates a combined evaluation approach.

3 Three Design Principles

ACES is governed by three principles: one evaluation contract across layers and agents, paired differential measurement, and developer-guided evaluation that preserves author intent.

  • Principle 1: Write once, evaluate everywhere: A single evaluation contract drives every evaluation layer and agent, supporting common evals.json datasets plus richer fixtures, BYOT tasks, and BYOG logic.
  • Principle 2: Differential measurement: Paired conditions hold the question, agent, model, task assets, grading policy, and support skills constant while changing only target-skill availability.
  • Principle 3: Developer-guided evaluation: Automated dataset generation bootstraps evaluation but is not an oracle; authors can override generated content through EVAL.md and custom evaluators.
  • Principle 3: Developer-guided evaluation: Author intent takes precedence when supplied, while ACES otherwise bootstraps a default dataset that authors can refine.

4 Live Agent Evaluation

ACES evaluates live agent behavior through paired with-skill and baseline runs, trajectory normalization, authoring assets, and multi-axis grading. It supports refinement from observed trajectories, portable sandbox execution, and separate isolation and group tests for content versus discovery and routing.

  • 4 Live Agent Evaluation: A live evaluation pairs with-skill and baseline conditions, keeps configured support skills fixed, grades each trajectory, and reports their difference as Skill Lift.
  • Evaluation assets: Evaluation assets include prompts, expected skills, reference answers, observable behaviors, fixtures, environment configuration, BYOT tasks, and BYOG graders.
  • Trajectory-Grounded Refinement: Trajectory refinement replaces aligned ground truth with the agent’s final answer and lifts observed tool-call sequences into expected behavior without overwriting EVAL.md directives.
  • Grading and review outputs: The default suite combines security, skill execution, skill efficiency, accuracy, goal accuracy, and behavior check, while BYOG adds domain-specific metrics.
  • Trajectory contract: ATIF standardizes trajectories as ordered steps containing sources, messages, tool calls with arguments, and observations for shared grading across harnesses.
  • Sandbox execution: ACES treats sandbox execution as a backend contract supporting containers, cloud environments, restricted scopes, specialized compute, hardware-in-the-loop workflows, and team-owned services.
  • Isolation versus group testing: Isolation mode measures content contribution with only the target skill, whereas group mode tests selection and execution alongside fixed supporting or decoy skills.

5 Evaluation-Native Skill Development

ACES treats evaluation as part of skill development: assets stay with skills, repository automation turns changes into review evidence, and evaluation runs can be change-aware and scheduled.

  • Evaluation-Native Skill Development: Evaluation-native development keeps evaluation assets with the skill so changes can be reviewed using document scans, security checks, and live paired agent evaluation.
  • Evaluation-Native Skill Development: ACES adapts continuous-evaluation practices to skills as first-class artifacts, connecting evaluation results with agent evolution and operation.
  • Testing modes: Figure 6 contrasts isolated testing with group testing alongside api-debugger, log-triage, config-validator, and release-planner.
  • Repository automation: Figure 7 depicts repository automation turning each skill change into review evidence before merge.
  • Evaluation assets: The evals/ directory functions as a skill test suite containing datasets, authoring intent, fixtures, execution policy, optional BYOT/BYOG assets, and custom environments.
  • Repository automation: Automation detects changed skills from the merge base, supports optional gating, and publishes per-skill scores and review artifacts or merge-request comments.

6 Empirical Evaluation

ACES evaluates skills through paired live-agent trials, revealing runtime effects that scans cannot observe. Across production cases, skills generally improved composite and outcome performance, while results varied by metric, harness, model, and skill.

  • 947 paired cases yielded mean composite Skill Lift of 0.2134, with 95% CI [0.1967, 0.2301].The cases covered 58 of 64 production skills across four primary harnesses.
  • Mean outcome-only lift was 0.1799, while composite lift was positive in 689 of 947 paired cases.Composite lift was zero in 171 cases and negative in 87.
  • Skill execution, behavior checking, and skill efficiency showed the largest gains, with lifts of 0.3263, 0.2983, and 0.2758.These process metrics capture workflow following, author-specified behaviors, routing, and tool use beyond final-answer correctness.
  • Harness means were positive but uneven: 0.3611 for OpenCode, 0.2904 for Claude Code, 0.1264 for Codex, and 0.0896 for Terminus-2.These are matched within-harness diagnostics, not an absolute model ranking or an unweighted estimate of the headline lift.
  • Stronger baselines can shrink measured Skill Lift even when absolute task performance remains high.A fixed-Codex model sweep demonstrates model–skill interaction in live evaluation.
  • Negative cases expose execution instability, truncated or meta-level responses, skipped verification, and extra tool calls without improved answers.Paired traces distinguish skills that were never discovered from skills that were discovered but misused.

7 Discussion and Limitations

ACES provides stronger paired evidence than static review but remains bounded by corpus coverage, judge uncertainty, model dependence, and environment constraints. The reported lift is therefore a setting-specific measure requiring continued validation and re-evaluation.

  • The results cover 145 skills and four primary harnesses with uneven coverage, and the corpus is skewed toward infrastructure-related categories.Replication across additional organizations, harnesses, and underrepresented skill categories would strengthen generalizability.
  • Live inter-judge agreement, human calibration, and judge-uncertainty propagation remain unmeasured.The reported three-judge spread applies to document-rubric scores rather than live trajectory grading.
  • Stronger models can reduce measured Skill Lift by raising baseline performance even when absolute task performance improves.The Codex model slice is a methodological demonstration, not a broad model-quality claim, motivating scheduled re-evaluation after updates.
  • VPN-inaccessible enterprise endpoints make goal_accuracy absolute values lower bounds and can distort absolute success metrics.Skill Lift remains informative because both paired conditions see the same network state.
  • Live paired evaluation is reserved for release candidates, high-risk skills, or reviewer-requested changes because it incurs token and deployment cost.Routing stress results support targeted release testing rather than per-edit execution in crowded workspaces.
  • The study reports confidence intervals and cluster checks, but richer subgroup intervals and formal tests for negative-lift classes remain future work.The raw-row audit also conflates timeouts with other no-score causes, so no separate timeout rate is reported.

8 Related Work

ACES extends existing skill, agent, judge, execution, and continuous-evaluation work by treating individual skills as executable artifacts evaluated on author-owned tasks. Its distinctive integration combines paired baselines, portable traces, and multi-metric grading.

  • Skill-centric evaluation: SkillsBench shows that skills matter but does not score individual skills as first-class artifacts, integrate with developer CI, or use author-owned tasks.ACES addresses these omitted dimensions through skill-specific evaluation assets and paired runtime trials.
  • Agent and tool-use benchmarks: Fixed-task agent and tool-use benchmarks treat skills as non-first-class, whereas ACES evaluates whether an arbitrary skill helps on its own tasks.This makes ACES complementary rather than a replacement for those benchmarks.
  • LLM-as-Judge methodology: ACES adapts LLM-as-Judge methods for goal_accuracy while adding trajectory-level checks for skipped steps, hallucinated tool use, and bypassed procedures.The approach responds to the limits of final-answer-only scoring.
  • Agent execution and trace infrastructure: ACES builds on Harbor’s sandboxed execution and ATIF-native interface rather than claiming that infrastructure as its own contribution.Its adapter and emitter translate assets, normalize trajectories, and feed a shared grading and Skill Lift layer across harnesses.
  • Industry practice: ACES specializes broader continuous-evaluation guidance to skill artifacts through an open, multi-tier pipeline spanning authoring, runtime trials, and grading.Industry documentation informs its dataset schema, generation buckets, and expected_behavior convention.

9 Conclusion and Future Work

ACES adds paired live evaluation to document scanning, showing how skills affect discovery, workflow execution, routing, tool use, and outcomes. Future work targets broader validation, longitudinal model-update studies, and stronger uncertainty analysis.

  • Across 947 scored paired cases, with-skill runs improved composite score by 0.2134 and outcome-only score by 0.1799 on average.The largest gains appeared in skill execution, behavior checking, and efficiency.
  • Paired traces also expose negative-lift cases involving routing overhead, incomplete execution, or harness-specific regressions that scans miss.ACES normalizes traces into ATIF, applies default and optional metrics, maps findings to stakeholder dimensions, and reports Skill Lift.
  • Future work includes longitudinal Skill Lift studies across model updates and broader cross-organization validation.The agenda also includes representative decoy or prerequisite selection, richer subgroup intervals, and formal tests for negative-lift classes.

A Metric Contract and Weighting

ACES defines six normalized runtime metrics and combines them with an inspectable equal-weight default, while also reporting outcome-only lift to separate task outcomes from process signals.

  • Equal 1/6 weights provide an inspectable diagnostic default, not an assertion that metrics are equally causal or valuable.Repositories may override the policy or add BYOG metrics while retaining every report component.
  • Outcome-only lift averages accuracy and goal accuracy to separate final task outcomes from discovery, routing, workflow, and tool-use signals.For the frozen headline, this view is (0.1431 + 0.2167)/2 = 0.1799.
  • Six default metrics are normalized to [0, 1] before aggregation.

B Evidence Audit and Sensitivity Checks

The evidence audit covers 947 scored paired cases and shows predominantly positive composite deltas, while routing stress increases latency and motivates selective live evaluation.

  • 689 of 947 composite deltas were positive, 171 were zero, and 87 were negative, with median 0.1717.All six default metrics were present for the scored paired cases.
  • Mean wall time rises from 258 seconds at one visible skill to 451 seconds at 20, while mean overall lift stays at 0.133–0.149.
  • At 50 visible skills, the with-skill pass rate is 0.55 and mean wall time is 1,290 seconds, versus 0.725 at one visible skill.The paper treats 50 visible skills as a routing/latency stress condition.

C Positioning Against Recent Skill Benchmarks

ACES positions itself as a combined static-and-live evaluation protocol that extends recent skill-centered work with fixed-support baselines, discovery pressure, and normalized multi-metric traces.

  • ACES combines fixed-support baselines, discovery pressure, author-owned assets, ATIF-normalized multi-metric traces, and static-versus-live production evidence.The paper claims neither pairing nor CI alone as its contribution.
  • Reviewer-requested cases that passed the scan gate exposed additional live evidence.Internal identifiers are generalized because raw traces contain internal paths and product details.
  • Static review and live evidence answer different questions, with document scores reported on a 0–100 scale and live deltas matched as with-skill minus baseline.
Loading 2608.20614v1…