Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills

• Outcome-only checks can accept correct-looking outputs reached through incorrect scope, skipped readbacks, or writes to the wrong table, risking downstream corruption and failures when inputs change.Source: §1 · p. 2
• 162 of 175 trials passing all applicable final numerical checks had another evaluator-detected deviation; under a broader baseline, 151 of 164 passing runs still violated a trajectory check.The framework’s 240-trial evaluation spans two specification variants, two harnesses, and three models.Source: §1 · p. 2
• The framework checks tool selection, arguments, execution order, and database integrity against independently computed ground truth, with a narrowly scoped LLM judge for semantically flexible arguments.Source: §1 · p. 2
• Runtime-resolved templates materialize expected scope, multiplicity, and values, and one registered suite was reused across the tested configurations.The contribution lists 104 Revenue cases and 80 Productivity cases.Source: §1 · p. 2

• API updates, specification revisions, model upgrades, and harness changes each introduce risks that outcome-only evaluation may not detect.Source: §2 · pp. 2–3
• Harnesses with similar outcome accuracy can exhibit distinct process-level deviation patterns, and a specification variant may be robust under one harness but reduce accuracy under another.Source: §2 · p. 3
• Persistent template test cases paired with process-level evaluation are proposed as the quality gate for evolving enterprise skills.Source: §2 · p. 3

• Prior work describes agent instability and reports that system-design and task-verification categories made up 63.1% of annotated failures in analyzed multi-agent traces.Source: §3 · p. 3
• Prior work combines deterministic checks, LLM judges, and human review as complementary signals; the paper similarly uses structural rules and targeted LLM validation, with a manual audit agreeing on 57 of 60 cases.Source: §3 · p. 3
• Generated test oracles can be erroneous: one cited study found 88.89% of erroneous HumanEval test cases stemmed from incorrect output oracles, while this framework computes ground truth through the same real APIs independently of the agent trace.The paper notes a separate risk that live state can change between execution and evaluation.Source: §3 · p. 3
• Process evaluation can reveal noncompliant paths hidden by final-output grading, and process supervision has outperformed outcome-only supervision in mathematical reasoning.Source: §3 · p. 3
• AgentEval uses a dependency-modeled DAG for step-level checks; this work adds reusable runtime-resolved templates and ground truth computed from live APIs independently of agent execution.The experiments establish reuse across configurations, not longitudinal persistence under API change.Source: §3 · pp. 3–4

• BVD allocates business value across applications using utilization data, and its persistent outputs are read by downstream skills, making errors consequential for the remaining pipeline.Source: §4 · p. 4
• Revenue Apportionment stages monitoring data, reads it back, allocates revenue, and writes host- and application-level tables; Productivity adds cost retrieval, resource identifiers, cost splitting, and explicit NULL semantics.Their suites contain 104 and 80 checks, respectively.Source: §4 · p. 4
• MD–TXT differences reflect full instruction revisions, not a controlled format change: Revenue TXT is substantially condensed, while Productivity TXT is modestly shorter and adds operational content.Both variants delegate deterministic arithmetic to tools.Source: §4 · p. 4

5 Continuous Evaluation Framework

• The five-stage pipeline runs independently of the agent session, with capture and reporting performed per run and ground-truth, test-materialization, and evaluation stages reused within each skill.Stages 1 and 5 are new for each run; Stages 2–4 reuse a template suite across evaluated specifications, models, and harnesses.Source: §5 · p. 4, Figure 1 · p. 5

5.1 Stage 1: Capture Input and Parse Tool-Call Trace

• A structured input log and harness execution log capture the run’s parameters and full tool-call trace.The trace records tool name, arguments, return value, and conversation turn index.Source: §5.1 · p. 4

5.2 Stage 2: Compute Ground Truth

• An independent script calls the skill’s real APIs to compute expected per-application values, while live-state changes between execution and evaluation remain a validity threat.Source: §5.2 · p. 5

5.3 Stage 3: Materialize Template Test Cases

• Runtime-resolved placeholders let the same parameterized template suite materialize expected calls and values for different runs, provided intended tool semantics remain unchanged.Materialization expands declarations into presence, argument, scope, execution, redundancy, and call-count checks, with dependency links between checks.Source: §5.3 · p. 5

5.4 Stage 4: Evaluate

• Evaluation separates final numerical outcomes, broader final-state checks, and trajectory checks, examining tool use, arguments, ordering, and scope.Programmatic checks handle candidate selection and structure; a narrowly scoped LLM judge addresses designated semantic variations and supplements, rather than replaces, the programmatic suite.Source: §5.4 · pp. 5–6

5.5 Stage 5: Report Root Failures with Cascade Attribution

• Dependency links group cascaded failures under designated roots to reduce the initial review set, but those graph-derived roots are not validated causal explanations.Source: §5.5 · p. 6

Skills and data

• Revenue and Productivity use 104 and 80 template cases respectively; Productivity adds cost retrieval and allocation, NULL handling, and a different persistence contract.Source: §6 · p. 6
• The study uses a real nine-application portfolio, but compares configurations within each skill because their trials use different date ranges and Productivity also queries cost APIs.Utilization is retrieved live; Productivity additionally queries Kubecost and Cloudability, so cross-skill differences are not interpreted under a shared data snapshot.Source: §6 · p. 6

Evaluation design

• The 2 × 2 × 2 × 3 design crosses skills, MD/TXT specification variants, Claude Code/Codex harnesses, and three models, with 10 trials per cell for 240 trials.Suite scores aggregate outcome and process checks and are not probabilities that entire runs are correct.Source: §6 · pp. 6–7

Statistical analysis

• Exploratory difference-in-differences intervals use 20,000 bootstrap resamples of ten runs per cell, while the broader final-state layer comprises seven checks per skill.Intervals are unadjusted for six comparisons and exploratory given the small per-cell sample; the study also reports trial-level means and standard deviations.Source: §6 · p. 7

Cross-Model and Cross-Harness Results

• 162 of 175 trials passing all applicable final numerical checks (92.6%; Wilson 95% CI: 87.7–95.6%) had another evaluator-detected deviation.All 120 Revenue trials passed the final revenue check and all 120 had at least one additional detected deviation; final-value checks alone would miss deviations in 67.5% of all trials.Source: §7.1 · p. 7
• Across the complete suite, 227 of 240 runs (94.6%; Wilson 95% CI: 91.0–96.8%) failed at least one check.The conclusion that outcome-correct trials also had deviations persisted under the reported external-error and unrecovered-error filters.Source: §7.1 · p. 8
• Among 162 outcome-correct but flagged trials, non-exclusive root families included data or numerical consistency (106), missing phase or required tool (66), allowlist violations (55), and redundancy or call-count violations (43).The counts can sum beyond 162 and do not assign severity; removing the no_irrelevant_tools check still left another failed check in all 162 trials.Source: §7.1 · p. 8
• 151 of 164 trials passing all seven final-state checks (92.1%; Wilson 95% CI: 86.9–95.3%) still violated a trajectory check.The rates were 109/109 for Revenue and 42/55 (76.4%) for Productivity; removing no_irrelevant_tools left the count unchanged.Source: §7.1 · p. 8
• Mean failed checks per run fell from 6.34 to 2.65 root failures under dependency attribution, a 58% reduction in items requiring initial diagnosis.The graph-derived roots are descriptive and were not independently validated as causal explanations.Source: §7.1 · p. 8
• Specification sensitivity depended on model, harness, and skill: interaction intervals excluded zero for all three Revenue comparisons and for GPT on Productivity, but not Productivity Opus or Sonnet.For Revenue, GPT-5.6-Sol shifted from −1.2 pp on Claude Code to +12.9 pp on Codex; the paired files do not isolate formatting from content.Source: §7.1 · p. 8
• Claude Code had higher Revenue TXT accuracy across models (86.8–95.7% vs. Codex 82.1–92.6%), while unrecovered errors were concentrated in Codex runs (38 vs. 5).All 20 Productivity Codex–Sonnet runs had an unrecovered error, so clean-execution effects for that configuration were not estimable.Source: §7.1 · p. 8
• Harness telemetry recorded $859.34 across 240 runs (mean $3.58, median $1.23), but these observational costs are not causal harness comparisons.Source: §7.1 · p. 9

8.1 Template Test Cases as Reusable Regression Artifacts

• One registered suite per skill covered all 24 evaluated configurations, and archived materializations showed the same check IDs across four Revenue and 16 Productivity expected-value states.The study does not test controlled input or API evolution or establish maintenance-free validity after semantic changes.Source: §8.1 · p. 9

8.2 Evaluation Surfaces Specification Gaps

• Evaluator flags exposed implicit policy choices in EC2 memory and cost allocation, database readback ordering, and EC2 call scope.The authors reserve “violation” for breaking an authored invariant; domain review must distinguish defects, benign alternatives, and specification ambiguities.Source: §8.2 · p. 9

8.3 Specification-Revision Sensitivity Across Harnesses

• Specification sensitivity could invert across harnesses: GPT-5.6-Sol was variant-robust on Claude Code for Revenue (−1.2 pp) but most sensitive on Codex (+12.9 pp).Paired MD/TXT files also differ in content, so the study cannot attribute effects to formatting itself; a specification validated on one harness should be re-evaluated after migration.Source: §8.3 · p. 9

8.4 Outcome Accuracy Does Not Establish Process Conformance

• Even with computation tools as the operational baseline, 162 of 175 numerically correct runs had another detected deviation, and 151/164 runs passing seven broader checks violated a trajectory check.Process-level regression tests add coverage beyond the chosen final state, while the operational severity of each deviation remains domain specific.Source: §8.4 · p. 9

9 Conclusion

• 162/175 runs with correct final numerical outcomes contained another evaluator-detected deviation; under the broader final-state baseline, 151/164 passing runs still violated a trajectory check.Dependency attribution reduced 6.34 failed checks to 2.65 roots per run on average.Source: §9 · p. 10
• Dependency attribution reduced the mean number of failed checks per run from 6.34 to 2.65 roots.Source: §9 · p. 10
• The findings support process-level regression tests as complements to final-state evaluation, with template reuse and harness-dependent sensitivity motivating re-evaluation as skills evolve.Source: §9 · p. 10

A Limitations

• The study evaluates one enterprise system and two related skills; generalization to other skill types, tool suites, or non-quantitative outputs remains open.Productivity extends Revenue with external cost sources, cost allocation, NULL handling, and a different persistence path.Source: A Limitations · p. 10
• Ten trials per cell limit power for small effects, and bootstrap interaction intervals are exploratory; two of six Productivity comparisons do not establish an interaction..Source: A Limitations · p. 11
• MD and TXT bundle differences in syntax, length, detail, and sometimes semantics, so observed gaps cannot be attributed causally to format or condensation alone.Productivity TXT adds exclusion, normalization, combined-cost-query, and CSV-export requirements.Source: A Limitations · p. 11
• The aggregate suite score omits failure severity and combines dependency-linked, non-independent checks, so small score differences need category-specific root-failure rates for operational interpretation..Source: A Limitations · p. 11
• The 83/240 runs with tool or harness errors may affect harness and variant contrasts; all runs were retained, while clean-run analysis is limited by zero-clean cells.External and unrecovered errors overlap, and excluding unrecovered-error runs removes all 20 Productivity Codex–Sonnet observations.Source: A Limitations · p. 11, Table 2 · p. 11
• Some evaluator checks encode workflow policy, and EC2-memory disagreements expose underspecified assumptions rather than proving that the oracle’s expected value is uniquely correct.The irrelevant-tool check is a root failure in 71/240 trials; removing it leaves the 162/175 headline result unchanged.Source: A Limitations · p. 11
• The 57/60 author-audit agreement on LLM-judged cases is not general judge accuracy; coverage was unstratified, reviewers were authors, and the failure-only rescue path was not separately characterized.Live-API ground truth can drift before deferred evaluation, and calculator-backed arithmetic does not eliminate semantic-judgment risk.Source: A Limitations · pp. 11–12
• Harness-recorded USD estimates are not reconciled provider invoices, so cost results describe these 240 executions rather than an intrinsic harness price advantage.Pricing, cache accounting, contracts, and expensive retries can differ across harnesses and models.Source: A Limitations · p. 12

B Cost and Runtime Results

• Claude Code averages $1.04 versus $6.12 for Codex; Claude-model cost ratios are 5.9–6.0× for Opus, 8.8–10.2× for Sonnet, and 1.0–1.4× for GPT.Ratios pool variants within each skill; Claude-model cache-read tokens comprise 90.7–92.9% of Claude Code tokens versus 0% on Codex, consistent with caching as a major accounting difference.Source: B Cost and Runtime Results · p. 12, Table 3 · p. 12
• Across all 240 runs, harness telemetry reports $859.34 total cost (mean $3.58, SD $4.50; median $1.23; range $0.45–$30.23 per run).These are recorder-estimated observational costs, not invoice-validated causal effects.Source: Table 3 · p. 12, B Cost and Runtime Results · p. 12
• Mean runtime for Claude Code/Codex is 416/238 seconds for GPT, 254/450 for Opus, and 284/344 for Sonnet.The Codex–Opus mean is inflated by one 7,152-second run; its median is 241 seconds.Source: Table 3 · p. 12, B Cost and Runtime Results · p. 12
• Table 4 reports mean recorded USD cost per run ± sample standard deviation for every cell in the 240-run design, with no missing or zero cost telemetry.Each cell summarizes 10 trials.Source: B Cost and Runtime Results · p. 12, Table 4 · p. 13