PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Strategy-Level RSI for Skill-Augmented Agents: Learning When to Reuse Skills from Execution Feedback

Shuang Guo, Xiaobo Sui, Fei Jiang, Ruixin Li, Manxiang Li, Rong Li, Ziyang Ou

arXiv:2609.26863v3cs.MA

TL;DR

A candidate Skill’s relevance does not establish that loading it will help in the current state. SkillApt uses matched WITH/WITHOUT executions to revise an external Skill-use policy while keeping the model, architecture, and Skills fixed. It reaches 91.3% task success while activating Skills on 26.3% of states, matching a strong zero-shot controller’s success at a lower activation rate.

  • Problem

    Once a candidate Skill is known, it remains unclear whether it is worth loading in the current state, since relevance or WITH-Skill success alone does not show marginal benefit.

    Source: §1 · p. 1, §3.3 · p. 3

  • Method

    SkillApt uses matched WITH/WITHOUT execution outcomes to estimate a candidate Skill’s conditional marginal utility and choose Load or Abstain.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    91.3% task success after 12 paired evidence states per Skill matches a strong zero-shot controller, while SkillApt activates Skills on 26.3% of states versus 98.8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Paired outcomes better support learning a Skill’s marginal utility than WITH success alone, which leaves the policy loading almost everywhere.

    Source: §8 · p. 12

    The primary experiment covers LogicBench and MedCalcBench and generalizes to held-out states of known Skills, not unseen Skills.

    Source: §7 · p. 12

Abstract

from arXiv · show

Long-running agents accumulate reusable Skills, but a Skill that is semantically relevant to a task is not necessarily worth loading in the current state. We study the applicability question that arises once a candidate Skill is known: should it be loaded in the current state? We propose SkillApt, which uses matched WITH/WITHOUT Skill executions on the same task state as persistent evidence, estimates the conditional marginal utility of the Skill, and chooses LOAD or ABSTAIN accordingly. The base model, agent architecture, and Skill contents stay fixed; only the external deployment policy changes. We call this constrained setting strategy-level recursive self-improvement (Strategy-Level RSI). On 20 Skills and 160 held-out states, as paired evidence accumulates, SkillApt's task success rises from 81.9% under a cold start to 91.3%, matching a strong zero-shot LLM controller; yet SkillApt activates Skills on only 26.3% of states, versus 98.8% for the zero-shot controller. A hard-candidate study shows that non-optimal Skills mostly leave correctness unchanged while raising execution cost, and occasionally cause correctness harm. An ablation shows that a history recording only WITH success makes the policy load almost everywhere, whereas paired evidence substantially improves selectivity. These results indicate that relevance is not applicability: the main effect of execution evidence is not to make the model stronger but to change how existing Skills are deployed, moving the system from near-always loading to selective reuse.

1 Introduction

The paper asks whether a known candidate Skill should be loaded in the current state, distinguishing applicability from retrieval relevance. SkillApt uses paired execution evidence to revise a fixed agent’s deployment policy, matching a strong zero-shot controller’s success while activating Skills on far fewer states.

1 Introduction

  • A relevant Skill may add cost or turn a correct answer wrong, so retrieval relevance and successful WITH-Skill execution do not establish current-state applicability.SkillApt addresses applicability after a candidate is known, not retrieval.Source: §1 · pp. 1–2
  • Strategy-level RSI changes an external deployment policy using persistent evidence while keeping the base LLM, agent architecture, and Skill contents fixed.The paper excludes unrestricted RSI, autonomous online self-modification, Skill evolution, and model self-training.Source: §1 · p. 1
  • 91.3% task success after 12 paired evidence states per Skill matches a strong zero-shot controller, while SkillApt activates Skills on 26.3% of states versus 98.8%.The primary experiment covers 20 Skills and 160 held-out states; cold-start task success is 81.9%.Source: §1 · p. 2
  • The paper frames applicability as a separate decision from retrieval and proposes SkillApt to choose whether an already-proposed candidate is worth loading.The introduction also reports that paired evidence keeps Skill exposure selective in ablations.Source: §1 · p. 2
  • SkillApt estimates a candidate Skill’s conditional marginal utility from matched WITH/WITHOUT executions and chooses Load or Abstain.The paired record includes both outcomes and a derived utility label; empirical claims concern a controlled offline protocol.Source: §1 · pp. 1–2

2 Related Work

Related work studies experience-driven agent improvement, Skill acquisition and evaluation, selective invocation, and feedback-based routing. SkillApt’s focus is narrower: revising a persistent reuse policy for fixed Skills and a fixed executor from matched outcomes.

2.1 Experience-Driven Agent Improvement

  • Experience-driven methods improve agents at memory, workflow, architecture, or code levels; strategy-level RSI instead changes the deployment policy while leaving those components fixed.Prior examples include verbal or distilled memory, induced workflows, searched designs, and modified programs.Source: §2.1 · p. 2

2.2 Skill Acquisition, Retrieval, and Evolution

  • Skill research spans acquisition, retrieval, optimization, and evaluation, whereas SkillApt concerns governing use of fixed artifacts rather than changing them.The related work includes Skill-centered evaluation and auditing as well as Skill optimization.Source: §2.2 · p. 2

2.3 Selective Skill Invocation and Skill Utility

  • Selective invocation methods learn preferences or encode applicability boundaries; SkillApt instead adapts a persistent external reuse policy from historical matched outcomes.Its budgeted BASM-inspired comparison evaluates one explicit-boundary implementation, not the full BASM system.Source: §2.3 · p. 2

2.4 Adaptive Routing and Calibration

  • Feedback-based routing selects among workflows or models, while SkillApt decides whether to expose a textual Skill to a fixed executor.ProbeCal calibrates uncertainty about tool effectiveness rather than learning a Skill reuse policy.Source: §2.4 · pp. 2–3

3 Method

The method treats Skill applicability as a deployment-policy decision: paired WITH/WITHOUT execution evidence revises whether a known candidate Skill is loaded while the model and Skill contents remain fixed. SkillApt stores matched outcomes and uses local evidence to choose Load or Abstain.

3.1 Strategy-Level RSI Setting

  • Strategy-level RSI revises an external Skill deployment policy from persistent execution evidence while keeping model parameters, agent architecture, and Skill contents fixed.The experiments use offline evidence-prefix replay and sequential replay, not unrestricted autonomous self-modification.Source: §3.1 · p. 3

3.3 Paired Execution Evidence

  • Matched WITH/WITHOUT executions on the same task state estimate a candidate Skill’s marginal utility; each pair is a descriptive contrast, not an identified causal effect.Only one matched pair per state is observed, leaving residual generation variability.Source: §3.3 · pp. 3–4
  • The evidence target prioritizes correctness changes, while tied-correctness executions receive cost labels based on tokens, latency, and calls.The multi-resource cost rule is an operational proxy, not a monetary cost model.Source: §3.3 · p. 5

3.4 Evidence Accumulation

  • SkillApt stores the state representation, both arm outcomes, and a derived utility label, indexed by Skill and executing model rather than storing execution trajectories.Held-out outcomes are excluded from the evidence store.Source: §3.4 · p. 5

3.5 Evidence-Conditioned Skill-Use Policy

  • The policy uses local, same-Skill evidence to decide whether to Load or Abstain; with no history, its prior selects Abstain.The estimator uses up to five highest-weighted neighbors, and its score is a smoothed evidence statistic rather than a calibrated probability.Source: §3.5 · p. 5, §3.6 · p. 5

3.7–3.9 Variants and Adaptation

  • The paper also evaluates semantic and fusion scoring as alternatives, a global logistic-regression baseline, and policy revision as evidence changes.More evidence has no monotonicity guarantee because local neighborhoods can change or expose conflicting effects.Source: §3.7 · p. 5, §3.8 · p. 6, §3.9 · p. 6

4 Experimental Setup

The evaluation asks whether paired execution evidence improves future Skill use, why relevant Skills should not always be loaded, and whether paired evidence is needed. It tests these questions on held-out task states with controlled Skill conditions and policy comparators.

4.1 Research Questions

  • The study evaluates whether historical paired evidence improves Skill use, why relevant Skills should not always be loaded, and whether paired outcomes are necessary.Threshold, cross-model, and lifecycle checks provide robustness and external-validity evidence.Source: §4.1 · p. 6

4.2 Tasks, Splits, and Executors

  • The primary study selects 20 Skills across LogicBench and MedCalc-Bench, using 12 evidence states and eight held-out states per Skill.The held-out task states, rather than unseen Skills, are the generalization target.Source: §4.2 · p. 6
  • Each primary-study state is executed with and without its associated candidate Skill under the same prompt shell and evaluator; adding the Skill does not force compliance.The primary experiment tests applicability for a known candidate and does not introduce retrieval errors into the adaptation test.Source: §4.2 · p. 6

4.3 Comparators

  • Comparators include a zero-evidence Always-Abstain policy, a zero-shot controller, paired-evidence SkillApt, and ablations including WITH-only and Global-5.The BASM-inspired D2 comparator uses evidence-derived natural-language boundaries within a 256-token budget and is not a full BASM reproduction.Source: §4.3 · pp. 6–7

4.4 Metrics

  • Offline selected-arm task success is the primary metric, alongside activation and mean selected-arm executor tokens; offline evidence-acquisition and controller overhead are excluded from tokens.Policy accuracy is additionally reported for RQ3 against a utility-derived target, excluding neutral states.Source: §4.4 · p. 7, Table 1 · p. 7

5 Results

Across four research questions, SkillApt uses paired execution evidence to selectively deploy Skills: it matches the strong zero-shot controller’s success while loading far less often. Results also show that irrelevant or redundant Skills can add cost, and that WITH/WITHOUT evidence supports selectivity better than WITH-only history.

5.1 RQ1: Can Historical Paired Execution Evidence Improve Future Skill Use?

  • 91.3% task success after 12 paired evidence states per Skill matches the strong zero-shot controller, while SkillApt loads on 26.3% of states versus 98.8%.Mean selected-arm tokens fall from 1,811.7 to 1,188.5; the primary comparison is on 20 Skills and 160 held-out states.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2 RQ2: Why Not Always Load a Relevant Skill?

  • On 199 complete triplets, hard candidates score 78.4% success versus 81.4% with No Skill, a difference that is not statistically significant.In 169/199 exposures, correctness is unchanged while execution cost increases; 12 exposures turn a correct answer incorrect, with six harms persisting on reruns.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 RQ3: Why Are WITH/WITHOUT Paired Evidence Necessary?

  • WITH-only history matches SkillApt’s 91.3% task success but loads on 99.4% of held-out states, versus 26.3% for SkillApt.Policy accuracy is 17.1% for WITH-only and 77.0% for SkillApt; Global-5 is close to SkillApt, but this experiment cannot separate state similarity’s contribution from evidence capacity.Source: §5.3 · pp. 8–9

5.4 Robustness and External Validity

  • The sequential replay’s gain over round-matched Cold-start rises from +2.5 pp in rounds 1–4 to +6.25 pp in rounds 9–12, without implying monotonic improvement or autonomous online RSI.Across 500 within-Skill orderings, cumulative gain remains positive in all permutations and late gain exceeds early gain in 78%; replayed states are evidence, not held-out states.Source: §5.4 · p. 9, Figure 4 · p. 10
  • At τ = 0.4, activation is 100% and selected tokens are 1,823.3 without exceeding τ = 0.5’s 91.3% task success; at τ = 0.6, success falls to 86.3%.The threshold sweep supports a selectivity–performance trade-off, not a theoretically optimal threshold.Source: §5.4 · p. 9, Figure 5 · p. 10
  • On matched sets of 10 Skills and 80 held-out states, GPT-5.6 Terra improves from 82.5% to 91.3% and Claude Sonnet 4.6 from 65.0% to 70.0%.Claude’s confidence interval spans zero, so this is directional replication rather than evidence of a stable cross-model gain.Source: §5.4 · p. 9, Table 3 · p. 10
  • On a fresh cohort, SkillApt reaches 87.1% success versus 84.2% for the zero-shot controller, but the +2.9 pp difference is not significant.SkillApt loads on 30.8% of states versus 95.4%; the cohort supports matching success at lower exposure, not higher success.Source: §5.4 · p. 10
  • Supplementary software-agent evidence shows that formerly relevant validators may no longer trigger on the current path, but it does not establish production-level longitudinal RSI.This evidence is not training data for SkillApt and does not demonstrate trajectory-based policy learning or autonomous online adaptation.Source: §5.4 · p. 10

6 Discussion

The discussion distinguishes improving an agent’s capabilities from improving how it deploys capabilities it already has. It argues that paired feedback helps attribute marginal benefit, while noting limits on evidence acquisition and stale evidence.

6.1 What Self-Improves

  • Strategy-level RSI changes the deployment policy through persistent evidence while keeping the model, architecture, and Skill text fixed.The zero-shot comparison reaches the same task success with or without historical evidence, so the reported change is in capability use rather than capability.Source: §6.1 · p. 11

6.2 Why Paired Feedback Matters

  • WITH-only success records can credit a Skill for tasks solvable anyway; paired feedback records the same state’s reference outcome and separates assisted success from marginal benefit.Deployment sees only the selected arm, while the study’s paired evidence came from separate offline execution; when acquiring such evidence is worthwhile remains open.Source: §6.2 · p. 11

6.3 Implications for Long-Running Skill-Augmented Agents

  • SkillApt does not learn the boundary between cost-positive and cost-negative ties, where the hard-candidate study locates most unnecessary exposure.Skill utility can change with the environment, validator, tool contract, or executing model, and detecting stale evidence was not evaluated.Source: §6.2 · p. 11, §6.3 · p. 11

7 Limitations

The evaluation and estimator are bounded by textual procedural Skills, paired execution outcomes, and evaluators that measure final-answer correctness. The study also limits its claims to held-out states of known Skills and does not establish amortized savings or online autonomous RSI.

Skill form and evidence form

  • The experiments mainly operationalize Skills as textual procedural context and train from matched outcomes, not full execution trajectories.Script-based, tool-based, and hard-constraint Skills remain future work; real-environment episodes are supplementary, not estimator training input.Source: §7 · p. 11

Evaluation and estimator

  • Correctness ties mean the evaluator detected no arm difference, not equivalent reasoning, completeness, robustness, or artifact quality.The estimator inherits defects the final-answer evaluator misses.Source: §7 · pp. 11–12
  • The primary experiment covers LogicBench and MedCalc-Bench and generalizes to held-out states of known Skills, not unseen Skills.The estimator is low-capacity, uses lexical hashing rather than a learned state encoder, and assigns all correctness ties the same target.Source: §7 · p. 12

Cost and statistics

  • Paired evidence and model-based boundary checks add offline costs; selected-arm token savings do not establish amortized total savings without a specified reuse horizon.Single observed pairs retain generation variability, and cost labels include latency noise.Source: §7 · p. 12

Scope

  • The scope comparison is not a full BASM reproduction, stronger-retrieval robustness is unresolved, and the cross-model replication is statistically inconclusive.Real-world evidence uses few fixtures, repeated runs, and in-sample trajectory mining; drift detection and online evidence-validity management were not evaluated.Source: §7 · p. 12
  • The evidence does not support online autonomous RSI, production improvement over calendar time, monotonic continual improvement, Skill evolution, or cross-model or full-BASM superiority.Efficiency effects differ across baselines and models.Source: §7 · p. 12

8 Conclusion

SkillApt revises a fixed agent’s external Skill-use policy using matched WITH/WITHOUT execution outcomes. In controlled experiments, paired evidence improves held-out success over cold start, matches a zero-shot controller at roughly a quarter of its activation, and makes reuse more selective than WITH-only history.

8 Conclusion

  • The study addresses whether a known candidate Skill should be used now, rather than whether to retrieve or create Skills.SkillApt uses matched outcomes to revise a fixed agent’s external Skill-use policy.Source: §8 · p. 12
  • Historical paired execution evidence improves held-out task success over a conservative cold start and matches a strong zero-shot controller while using roughly one quarter of its activation.Source: §8 · p. 12
  • Non-optimal Skills mostly add execution cost without changing correctness and occasionally cause correctness harm.This supports the distinction between a Skill’s relevance and its applicability in the current state.Source: §8 · p. 12
  • Paired WITH/WITHOUT outcomes better estimate marginal utility than WITH-success history alone, which makes the policy load almost everywhere.The paired-evidence policy revises future use while model weights and Skill contents stay fixed.Source: §8 · p. 12

A Experimental Protocol and Reproducibility

The protocol freezes the experimental conditions and isolates held-out task states while evaluating policy decisions and executor outcomes. It specifies the controller, representation, cost measures, and implementation qualifications needed to interpret replication and comparison results.

A Experimental Protocol and Reproducibility

  • The paper’s SkillApt and SkillApt-Fusion correspond one-to-one to artifact identifiers SkillApt-E and SkillApt-E+Sem, respectively.The artifact identifiers are internal condition names, not different methods.Source: A Experimental Protocol and… · p. 14
  • The primary design uses 20 Skills and 20 fresh tasks per Skill, splitting 12 states for evidence and eight for held-out evaluation.Skills are selected deterministically across two domains; held-out states, not unseen Skills, are the generalization target.Source: A Experimental Protocol and… · p. 14
  • The frozen representation hashes whitespace unigrams and adjacent bigrams into 256 dimensions; the five highest-weight neighbors contribute to Eq. 10.Cosines are truncated below at zero, and all correctness-tied classes map to 1/2.Source: A Experimental Protocol and… · p. 14
  • Model identities are provider-reported, and infrastructure failures do not generate utility labels; the frozen analysis also retains disclosed implementation differences.The main endpoint uses all 12 evidence items, while smaller-prefix comparisons do not control identical evidence membership.Source: A Experimental Protocol and… · p. 14
  • The zero-shot controller sees only the current task and candidate Skill text, with no execution history or paired outcomes.Its frozen prompt requires exactly LOAD or ABSTAIN; it uses 160 primary calls and was not tuned on outcomes.Source: A Experimental Protocol and… · pp. 14–15
  • Mean selected tokens exclude offline evidence acquisition and boundary-checking costs, while the primary success interval resamples the 20 Skill clusters.The fixed cost proxy includes prompt and completion tokens and measured successful-request latency; the reported cluster-bootstrap interval covers a specified contrast within a model.Source: A Experimental Protocol and… · pp. 14–15

B.1 Full Statistics for the Main Results

The full statistics report improved success over cold start and parity with the zero-shot controller, alongside selective deployment and trade-offs in policy-label accuracy and token use. Hard-candidate results do not establish aggregate degradation, though observed harms and screened confirmations motivate caution.

B.1 Full Statistics for the Main Results

  • +9.4 pp over cold start corresponds to exact McNemar p = 6.10×10−5; 15 states improve, none worsen, and five Skills improve while 15 are unchanged.Policy-label accuracy falls from 83.6% to 77.0%, mean selected tokens rise from 856.3 to 1,188.5, and all three correctness-harmful states remain avoided.Source: §B.1 · p. 15
  • SkillApt and the zero-shot controller have a paired task-success difference of zero; all 116 action disagreements switch zero-shot Load to SkillApt Abstain, 97 on tied cost-negative states.Mean selected tokens are 1,188.5 versus 1,811.7, before the controller’s own 1,900.9 tokens per state.Source: §B.1 · p. 15
  • Across 400 paired states, 297 are tied cost-negative and 41 are correctness-positive.The 240 evidence states include 23 correctness-positive and seven correctness-harmful pairs; the 160 held-out states include 18 and three, respectively.Source: §B.1 · p. 15

B.2 Full Hard-Candidate Statistics

  • Hard candidates differ from No Skill by −3.02 pp and from Gold by −5.03 pp, with neither aggregate degradation established at the 0.05 level.Correctness benefit occurs in six tasks, while initial correctness harm occurs in 12/199 (6.0%).Source: §B.2 · p. 15
  • Eight of 12 initial hard-candidate harms involve a rank-one non-gold candidate; eight tasks show No Skill wrong, Gold correct, and Hard wrong.Per-domain harm counts are too small for more than descriptive use.Source: §B.2 · p. 15
  • Six of the 12 initially harmful tasks still show lower hard-candidate correctness under confirmation, but screening only initial harms cannot estimate persistent-harm prevalence across all 199 tasks.Small per-domain counts and stochastic reversals motivate caution about extrapolation.Source: §B.2 · p. 15

B.3 Ablations and State Conditioning

The ablations compare SkillApt with explicit-boundary, global-evidence, WITH-only, and learned-applicability controls. They show that paired evidence supports selective deployment, while results against the boundary and global controls have specific scope limits.

B.3 Ablations and State Conditioning

  • +5.6 pp in task success and +13.2 pp in policy accuracy for SkillApt minus D2 at the full budget; this comparison does not establish superiority over full BASM.SkillApt activates on 42 rather than 53 states, and both systems avoid all three observed correctness harms.Source: §B.3 · p. 15
  • WITH-only success leaves task success unchanged at 146/160 but lowers policy accuracy from 117/152 to 26/152, with activation on 159/160 states.Among 204 evidence states jointly correct with and without the Skill, 168 were cost-negative; a positive WITH-only label conflates marginal-utility categories.Source: §B.3 · p. 16
  • SkillApt minus Global-All is +0.6 pp in task success and +13.2 pp in policy accuracy, while Global-5 differs by +1.9 pp and +1.3 pp, neither significantly.The global controls differ in effective evidence capacity, so the results support a more selective operating point but do not isolate state conditioning as uniquely necessary.Source: §B.3 · p. 17

B.4 Learned-Applicability Baseline Details

  • The offline logistic-regression baseline reaches 134/160 task success, 32/160 activation, and 115/152 policy accuracy, and loads on 9 of 25 utility-positive held-out states.The result shows only that this specific global linear baseline does not surpass SkillApt; its modeling family and lexical-hashing features differ, and it uses 225 training states across two domains.Source: §B.4 · p. 17

B.5 Sequential RSI Replay Details

The sequential replay tests SkillApt using only earlier same-Skill evidence, then compares its selected-arm outcomes with fixed references. The replay shows selective operation and a positive gain over round-matched Cold-start across tested orderings, though late-versus-early improvement is not deterministic.

B.5 Sequential RSI Replay Details

  • Each replay prediction uses only earlier same-Skill states, excludes held-out evidence, and records the selected arm before appending the paired outcomes and target.Because the replay still reads the frozen counterfactual arm after each decision, it is prequential replay, not production online learning.Source: §B.5 · p. 17
  • Sequential SkillApt achieves 221/240 (92.1%) task success with 56/240 (23.3%) activation, versus Cold-start at 211/240 (87.9%) and Always Load at 227/240 (94.6%).Its mean selected tokens are 1,140.5, compared with 866.5 for Cold-start and 1,806.5 for Always Load; the full-evidence reference is an upper reference, not an oracle or proven optimum.Source: §B.5 · pp. 17–18
  • SkillApt's gain over Always Load is at most zero in every round, with its value instead reflected in 36.9% fewer selected tokens than Always Load.Cold-start also improves from 69/80 to 71/80 across early and late rounds, so raw early-to-late success does not isolate SkillApt's gain.Source: §B.5 · p. 18
  • SkillApt's round-matched gain over Cold-start is positive in all 500 within-Skill orderings, but late gain exceeds early gain in only 390 (78%).The late-minus-early gain flattens or reverses in about a fifth of orderings; the frozen order is at the 76th percentile of cumulative success and 61st percentile of late-minus-early gain.Source: §B.5 · p. 18

B.6 Cross-Model Diagnostics

The diagnostics examine model agreement, replication on fresh states, retrieval-based comparisons, and token-only cost labels. They support selective reuse in these settings while bounding claims about transfer, retrieval strength, and generality.

B.6 Cross-Model Diagnostics

  • Terra's 95% CI is [+3.75, +15.00] pp and Claude's is [−3.75, +13.75] pp; sparse model-specific denominators preclude interpreting percentages as population safety rates.Claude activates all five of its correctness-harmful states at full budget (0/5 avoided), and its mean selected tokens rise from 759.1 to 1,222.2.Source: §B.6 · p. 18
  • WITH correctness agrees on 66/80 states and WITHOUT correctness on 64/80, while five-class utility agrees on 54/80 and binary targets change across models on 17 states.The agreement statistics are descriptive and do not demonstrate that a policy trained for one model transfers to another.Source: §B.6 · p. 18

B.7 Fresh-Cohort Replication Details

  • On the 240-state fresh cohort, SkillApt exceeds cold start by +5.0 pp, while its difference from the zero-shot controller is +2.92 pp with intervals spanning zero.The results are descriptive matched results on one executor, supporting the direction of the Phase A comparison rather than a superiority claim.Source: §B.7 · p. 19, Table 12 · p. 19

B.8 Retrieval-Based Confirmatory Study

  • SkillApt and BM25 Top-1 both achieve 83.8% observed accuracy, while activation falls from 100.0% to 31.5% and mean selected tokens from 4,782 to 1,231.The paired accuracy difference is zero, not statistical equivalence; the preregistered stronger-retriever condition was not established.Source: §B.8 · p. 20

B.9 Token-Only Cost Sensitivity

  • With token-only labels, all 181 Hard Candidate correctness ties increase tokens, and SkillApt avoids 109/136 cost-negative states on Phase A held-out data.The direction agrees with and slightly strengthens the multi-resource taxonomy, but none of the reported cost-positive ties remains cost-positive under tokens alone.Source: §B.9 · p. 20

C Real-World Software-Agent Lifecycle Evidence

The lifecycle evidence combines mined episodes with small controlled software-agent fixtures. The fixture results are local directional evidence, while the larger temporal replay did not reach outcome evaluation.

C Real-World Software-Agent Lifecycle Evidence

  • The 76 mined episodes came from two applications, with miner rules and gold labels developed on the same samples, so miner accuracy is not out-of-sample generalization.The controlled component used nine fixtures, two conditions, and three repetitions, totaling 54 designed runs.Source: C Real-World Software-Agent Lifecycle… · p. 20
  • Dataset-contract repair retains 100% success with about 18% lower median tokens, while global-import cleanup improves success from 80% to 100% with about 31% lower median tokens.These estimates come from 23 treatment-attributable dataset-contract runs and 30 global-import runs; per-repetition spread can reach 133% of a fixture/arm median.Source: C Real-World Software-Agent Lifecycle… · pp. 20–21, Table 13 · p. 21
  • The temporal-replay preparation stopped before any outcome API call because required checkpoints, evaluators, harness verification, Skill hashes, and tool-service controls were unavailable.No temporal task-success result or production-level longitudinal RSI is claimed.Source: C Real-World Software-Agent Lifecycle… · p. 21

Related papers

Loading 2609.26863v3…