PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Blocks: How Should an Agent Load Its Skill? A Caching-Correct Comparison of Pre-load, On-Demand Tool-Loading, Progressive Disclosure, and Hybrid

Hironobu Nakasuji

arXiv:2608.14943v1cs.AI

TL;DR

Agent skills are often loaded in full even when requests need only part of them, raising token costs. This paper compares content-preserving loading methods with cache-correct accounting and finds that no single method wins universally.

  • Problem

    Monolithic skills couple broader coverage to longer prompts, motivating evidence on selecting only the skill content each request needs.

  • Method

    The study compares conditional skill-block selection with reference, hybrid, and full loading while reporting raw, new, and cache-discounted multi-turn input.

  • Results

    No universal winner emerges: hybrid leads content-parity single-turn cells, whereas Skill Block and hybrid lead for large compressible multi-turn skills, with small always-needed procedures near parity.

  • Takeaways & Limitations

    Conditional loading is most useful when large portions of a skill are unnecessary on each turn, while packaging choice should follow the usage regime and coverage requirements.

  • Takeaways & Limitations

    Non-significant outcome tests do not establish equivalence or non-inferiority, and some primary outcome analyses were exploratory.

Abstract

from arXiv · show

Agent skills are often injected in full on every request, increasing token cost. We compare four content-preserving loading methods: Full, Skill Block, Reference, and Hybrid. Across SearchQA, SpreadsheetBench, ALFWorld, ScienceWorld, and SynthProc, we measure token usage using raw input for single-turn tasks and cache-correct effective input for multi-turn tasks. Results show no universal winner. Hybrid reduces input by 27.4% on SearchQA and 39.8% on SpreadsheetBench. On large multi-turn skills, Skill Block and Hybrid achieve substantial reductions, reaching 62.5% and 52.8% on ScienceWorld and 73.0% and 66.6% on SynthProc. ALFWorld shows smaller gains because procedures are short and repeatedly needed. Paired outcome tests detect no quality differences, though they do not establish equivalence. Overall, conditional loading is most beneficial when large portions of a skill are not needed on every turn.

1. Introduction

The section frames skill loading as a content-preserving deployment problem: monolithic skills couple broader coverage to longer prompts, motivating comparison of four loading mechanisms with caching-correct accounting. It presents a regime-dependent evaluation in which Hybrid helps single-turn tasks, Skill Block and Hybrid help large multi-turn skills, and no universal loading rule emerges.

  • Motivation: Monolithic skills concatenate all procedures to every prompt, coupling broader coverage to higher per-call prompt cost even when requests need only one procedure.The section distinguishes this loading problem from compressing skill content: the source skill remains available, but optional material need not be loaded immediately.
  • Loading mechanisms: Four content-preserving packagings are compared: Full, Skill Block, Reference, and Hybrid.Full sends all blocks; Skill Block keeps a core in context and tool-loads optional blocks; Reference uses a catalog and requested files; Hybrid uses stubs and fetches complete blocks when needed.
  • Loading mechanisms: Selection benefits depend on unused optional text, loading round-trips, and the amount of fixed context resent.The framing treats prompt-block selection as analogous to conditional computation, while retaining the skill’s own text verbatim rather than retrieving from an external corpus.
  • Evaluation: Cache-correct accounting reports raw input, new input, and discounted cache-read effective sensitivity for multi-turn conversations; raw input remains primary for single-turn comparisons.Provider input_tokens can include cache-read tokens, so counting all reported input as newly processed can misstate multi-turn cost.
  • Evaluation: The evaluation spans SearchQA, SpreadsheetBench, ALFWorld, ScienceWorld, and SynthProc across differing skill sizes and turn structures.The contributions include formalizing the unused-mass versus overhead trade-off and applying caching-correct accounting to the four-mechanism comparison.
  • Results: Hybrid is best in the two single-turn cells, Skill Block and Hybrid lead for large compressible multi-turn skills, and small always-needed multi-turn procedures show near parity.These are paired within-benchmark findings and do not establish equivalence, reliability, or a universal loading rule.

2. Related Work

Related work spans reusable skill infrastructure, prompt and pipeline optimization, conditional context selection, compression, and broad agent evaluation. This paper differs by keeping skill content fixed while studying when text is loaded, using verbatim selected blocks and benchmarks that vary optional skill content and repeated context footprint.

  • Skills and optimization: Agent Skills and MCP support reusable instruction documents and context delivery, while SkillOpt, APE, DSPy, TextGrad, and GEPA optimize skill content, prompts, or pipelines.The paper frames its intervention as orthogonal to these optimization methods: content remains fixed while loading timing changes.
  • Conditional context and compression: Skill loading routes inputs to subsets of text through model-mediated semantic choice, unlike trained sparse-MoE gates or usually lossy prompt compression.Selected skill blocks remain verbatim, and compression could be applied after selection.
  • Agent evaluation: Evaluation spans context-dominated QA, spreadsheet manipulation, text embodiment, interactive scientific simulation, and controlled per-step knowledge gating in SynthProc.These benchmark choices expose optional skill-text volume and whether an always-on footprint recurs over turns; SynthProc complements rather than substitutes for natural tasks.

3. Method

The method partitions skills into dependency-aware blocks and compares four content-preserving loading arms with distinct context, tool, and disclosure mechanisms. It also defines cache-correct input accounting and distinguishes dynamic loading from a non-parity static reduced-skill alternative.

  • Skill representation: Skills are partitioned into named, dependency-aware blocks whose complete rendering reproduces the monolithic skill, while core blocks remain always available.Blocks include descriptions, default flags, dependencies, priorities, and tags; already-present dependencies are not duplicated.
  • Loading arms: Full pre-loads the complete skill; Skill Block exposes a light core with schema-based block loading; Reference uses catalog selection without a tool schema; Hybrid adds compact stubs with optional full-block fetches.Skill Block may require a tool round-trip, Reference normally requires a selection round, and Hybrid can avoid fetching when its quick-reference view suffices.
  • Loading arms: Hybrid tiering varies by benchmark: some use compact stubs followed by complete blocks, while historical ALFWorld and ScienceWorld use lean hints and pointers before optional full references.Both are treated as progressive two-stage disclosure, but their tool-call counts are not treated as the same intervention.
  • Comparison limitation: Historical original8 is a separately run static reduced skill that omits six blocks, lacks a loader, differs from the dynamic arms’ core blocks, and does not establish content parity.It cannot show coverage of unobserved deployment demand.
  • Cost accounting: The accounting model weighs always-on core, optional-block load probabilities, fixed overhead, task context, schema tokens, and loading-interaction turns against savings from unused optional content.In multi-turn settings, persistent prompt and schema footprints recur; the expression is an accounting model and post-hoc engineering heuristic, not a learned selector or validated universal threshold.
  • Cost accounting: Multi-turn tables use effective input = new input + d · cache read with d = 0.1, while single-turn experiments primarily use raw input.Generated tokens are additionally incorporated through a provider-neutral total-cost sensitivity rather than a billing claim.

4. Experimental Setup

The experiments compare four content-preserving loading mechanisms on paired benchmark items, using raw input for SearchQA and SpreadsheetBench and effective input for stateful tasks. The setup reports benchmark-specific outcomes and emphasizes telemetry, endpoint, and inferential limitations.

  • Endpoint and telemetry: Most primary experiments used a GitHub Copilot production endpoint configured with GHCP_LEAN_SYSTEM=1 and reported as gpt-5.5, so absolute scores apply only to that endpoint.Historical logs do not independently confirm every model identifier returned by the provider.
  • Statistical interpretation: Paired exact McNemar tests found no detected outcome difference, but the exploratory tests do not establish non-inferiority or equivalence.Across nine principal gpt-5.5 accuracy comparisons, every Holm-adjusted p-value is 1.0 under the corrected ScienceWorld denominator.
  • Benchmarks: The primary benchmarks span 1,400 SearchQA single-turn cases, 276 token-complete SpreadsheetBench cases, 84 ScienceWorld multi-turn episodes, and 40 SynthProc multi-turn tasks.ScienceWorld uses an approximately 6K skill with 16 procedures, while SynthProc uses an approximately 10K, 45-operation library, with 44 operations exercised.
  • Evaluation design: Comparisons are paired on identical items, with raw input primary for SearchQA/SpreadsheetBench and effective input primary for ALFWorld, ScienceWorld, and SynthProc.Stateful turn usage is aggregated per episode, and medians are primary because long trajectories skew means.
  • Outcome measures: Outcome metrics remain native to each benchmark: SearchQA answer match, SpreadsheetBench workbook pass plus separately labeled cell accuracy, ALFWorld/SynthProc completion, and ScienceWorld binary completion plus progress.The four-arm comparison is lossless; static original8 changes content availability and is outside it.

5. Results

Results show no universal loading winner: conditional methods help most when skills are large and much content is unused, while short procedures yield limited savings. Paired tests found no quality differences, but do not establish equivalence or rule out regressions.

  • Single-turn results: 27.4%: Hybrid reduced SearchQA raw input to 4,016 versus 5,535 full tokens, while hard accuracy was .845.Hybrid fetched only 0.02 blocks per query, or 1.4% load-anything.
  • Single-turn results: 39.8%: Hybrid reduced SpreadsheetBench raw input to 7,506 versus 12,461 full tokens; Skill Block and reference achieved 35.6% and 31.7% reductions.Hard workbook pass was .800 full, .811 Skill Block, and .807 hybrid/reference, with no paired outcome difference detected.
  • Multi-turn results: 12.55%: Skill Block reduced ALFWorld effective input to 5,568.65 versus 6,368 full tokens, while every arm completed 42/42 episodes.Hybrid reduced input by 3.18%, but its saving was not statistically established under the aligned estimand.
  • Multi-turn results: 62.47% and 52.80%: Skill Block and hybrid reduced ScienceWorld effective input to 10,061 and 12,654 versus 26,810 full tokens.Success/progress were 62/84/.916 for Skill Block, 58/84/.902 for hybrid, 50/84/.833 for reference, and 57/84/.901 for full.
  • Multi-turn results: 73.0% and 66.6%: Skill Block and hybrid reduced SynthProc effective input to 5,837 and 7,221 versus 21,617 full tokens, with 40/40 completion in all arms.Raw counts can misrank arms because cache reads comprise 74–94% of raw multi-turn input.

6. Discussion

The discussion distinguishes single-turn overhead from repeated multi-turn footprints, while emphasizing that loading benefits depend on unused skill coverage. It also narrows the claims because selection, accounting, deployment, and model-transfer limitations remain.

  • Selection and optimization: Loading and content optimization address different levers, while selection quality is inferred indirectly and lacks labeled routing precision/recall evaluation.SpreadsheetBench’s execution-feedback loop can repair imperfect selection, so outcomes do not directly establish routing quality.
  • Single-turn and multi-turn cases: 1.4% SearchQA fetch rate makes Hybrid attractive in single-turn requests because its stub can prevent interaction overhead.In stateful interaction, repeatedly resubmitted always-on footprints make reducing a mostly unused procedure library more consequential.
  • Measurement limitations: Effective accounting corrects raw-token distortions from resubmitted prefixes, but provider billing, latency, reasoning tokens, and cache attribution remain unmeasured.The correction is evaluated under an explicit discount sensitivity and cannot recover omitted costs.
  • Deployment interpretation: Historical original8 is cheaper on the measured SpreadsheetBench split but permanently discards six procedures and has not been shown to preserve performance under shifted or broader requests.Therefore, the comparison does not refute content parity or demonstrate that dynamic routing is necessary.
  • Configured-model reruns: Configured-model reruns reproduce large effective-input savings for the on-demand pair, but their internal ordering changes or is nearly tied.This supports evaluating both mechanisms rather than asserting a stable model-specific rank or transfer beyond the tested provider.

7. Limitations & Threats to Validity

The study’s validity is limited by endpoint and production controls, exploratory outcome inference, synthetic and static probes, author-defined routing, and accounting assumptions. ScienceWorld analyses also involve incomplete historical records and temporally spliced repairs, while the post hoc cost gate does not support a universal size rule.

  • Endpoint and production scope: Results rely on configured endpoints and uncontrolled production behavior, so they provide paired within-benchmark comparisons rather than provider-attested isolation, cross-vendor replication, or public-model reproducibility.The primary endpoint is reported as GHCP gpt-5.5; transfer checks use gpt-5.6 without retained provider-returned model IDs, while system prompts and decoding are uncontrolled.
  • Outcome inference: Non-significant McNemar tests and token confidence intervals do not establish equivalence or non-inferiority, and reused development cases make the combined SearchQA and SpreadsheetBench outcome family exploratory.Token intervals are per-comparison rather than simultaneous family-wise intervals.
  • Probe and control validity: SynthProc isolates controlled token mechanics, SECRET KEY refusal loops confound gpt-5.6 reliability evidence, and historical original8 lacks content parity and non-inferiority support.Original8 is separately timed, differs from the dynamic core, and does not demonstrate coverage beyond the measured SpreadsheetBench split.
  • Selection and prompt scaffolding: Author-chosen block boundaries and unmeasured routing correctness limit interpretation, while accuracy is a weak routing proxy when feedback can recover errors.The mechanisms also require different instructions and catalogs, and SearchQA shows that guidance can materially change load behavior.
  • Accounting and execution assumptions: Accounting uses d=.1 as a sensitivity reference rather than internal GHCP billing, leaves schema-cache attribution unmeasured, and omits unobserved historical reasoning tokens and measured latency.ScienceWorld’s primary n=84 analysis excludes six arm-specific abort IDs from an incomplete historical run; its n=82 sensitivity and repaired matched n=90 artifact introduce additional exclusions and temporal splicing.
  • Gate status: The cost gate explains observed signs post hoc rather than predicting them prospectively, and the hybrid exception precludes a universal size rule.Appendix E further discusses temporal drift, cache writes, and guidance implementation implications.

8. Conclusion

Conditional loading can decouple coverage from per-query context cost, but the appropriate packaging depends on the operating regime.

  • Hybrid is the best content-parity option in two single-turn cells.
  • Large compressible multi-turn skills favor the Skill Block/hybrid pair.
  • Small always-needed multi-turn procedures are near parity under total-cost sensitivity.

Appendix A: Implementation and Reproducibility … Appendix B: Full gpt-5.5 Benchmark Tables and Paired Intervals

The appendices document endpoint caveats, reproducibility artifacts, rendering mechanics, optional multi-turn sections, and benchmark-specific skill packaging. They specify how Full, Skill Block, Reference, and Hybrid expose content across the evaluated tasks.

  • A.1 Endpoint, artifacts, and protocol: Endpoint artifacts record gpt-5.5 for regime-map runs, explicitly configure gpt-5.5 for targeted ScienceWorld replacements, and configure gpt-5.6-terra for transfer runs.Historical artifacts do not independently attest all provider-returned model and reasoning settings; transfer runs retain no provider provenance.
  • A.1 Endpoint, artifacts, and protocol: Repository READMEs document commands, file layouts, harness locations, retained result bundles, and analysis-script usage.SpreadsheetBench correction replay additionally requires omitted prediction workbooks and trajectories, while the corrected bundle preserves hashes and comparison evidence.
  • A.2 Skill format and dependency rendering: YAML front matter and EJS block delimiters define skills, while rendering expands defaults, requested blocks, and prerequisites not already present.Full rendering reconstructs source content; Skill Block registers a catalog-bearing loader, Reference exposes a text catalog, and Hybrid auto-generates a one-line stub.
  • A.2 Skill format and dependency rendering: Optional steering is injected once and execution-budget is rendered per turn with {used}/{max}/{remaining}, without changing the loading mechanism.These sections are presence-gated and were not separately ablated.
  • A.3 Per-benchmark packaging: SearchQA uses six blocks and approximately 1,966 full tokens, whereas SpreadsheetBench uses 14 blocks and approximately 8,116 full tokens.SearchQA’s Hybrid stub is approximately 385 tokens; SpreadsheetBench’s dynamic CORE_ BLOCKS are approximately 3,513 tokenizer-proxy tokens, and its Reference overview is approximately 526 tokens.
  • A.3 Per-benchmark packaging: ALFWorld has six approximately 99-token walkthroughs and approximately 1,087 full tokens, while ScienceWorld has 16 approximately 289-token procedures and approximately 6,097 full tokens.Reference reads full walkthroughs after selection; retained truehybrid exposes a selected hint before an optional full reference.

B.1 Single-turn results (raw input) · B.2 Multi-turn results (median raw / new / effective)

Single-turn raw-input results show hybrid and Skill Block can reduce tokens substantially versus full without detected outcome differences, while multi-turn effective-input reductions are largest for ScienceWorld and SynthProc. ALFWorld shows smaller savings, and reported paired tests do not establish equivalence.

  • B.1 Single-turn results (raw input): −27.4% versus full: SearchQA hybrid uses 4,016 raw input tokens, compared with 5,535 for full.Reference increases raw input by 10.4%, while Skill Block increases it by 48.4%.
  • B.1 Single-turn results (raw input): −39.8% versus full: SpreadsheetBench hybrid uses 7,506 raw input tokens, compared with 12,461 for full.Skill Block reduces raw input by 35.6%, while reference reduces it by 31.7%.
  • B.1 Single-turn results (raw input): Hybrid is cheaper than full on 1,380/1,400 SearchQA cases, while its block-loading rate is 1.4% versus 75% for Skill Block.Hybrid averages .02 fetches, whereas Skill Block averages 2.05 fetches.
  • B.1 Single-turn results (raw input): No detected outcome difference: Skill Block versus full has McNemar discordances of 25/26 with exact p=1.0 on SearchQA.On SpreadsheetBench outcome-clean cases, static original8 has McNemar p-value 1.0 versus each dynamic arm and .678 versus full, with temporal-drift and treatment-mismatch caveats.
  • B.2 Multi-turn results (median raw / new / effective): −12.55% effective input: ALFWorld Skill Block reduces the aggregate median versus full, while hybrid achieves −3.18%.Skill Block is cheaper on 38/42 episodes and hybrid on 34/42.
  • B.2 Multi-turn results (median raw / new / effective): −62.5% effective input: ScienceWorld Skill Block reduces the aggregate median versus full, while hybrid achieves −52.8%.The corresponding paired-median-difference estimates are −15,322 and −13,252 tokens.
  • B.2 Multi-turn results (median raw / new / effective): −73.0% effective input: SynthProc Skill Block reduces the aggregate median versus full, with a paired-median-difference estimate of −15,871 tokens.SynthProc full has an aggregate median effective input of 21,617 tokens, versus 5,837 for Skill Block.
  • B.2 Multi-turn results (median raw / new / effective): 81/84 primary ScienceWorld episodes are cheaper with Skill Block, 73/84 with hybrid, and 69/84 with reference.Sensitivity reductions are 62.05%/55.85%/25.57% at strict n=82 and 62.85%/50.84%/25.38% at matched n=90, with reference including zero.

Appendix C: Accounting and Robustness Details … D.2 Real ScienceWorld configured as gpt-5.6

The appendices show that cache-correct accounting materially changes cost comparisons, with conditional loading reducing effective input in multi-turn benchmarks and configured-model evaluations while outcome tests detect no significant ScienceWorld difference. They also document cache-policy sensitivity, statistical limitations, evaluator-isolation requirements, and robustness results for SynthProc and real ScienceWorld.

  • C.1 Cache fractions, means, and output sensitivity: ScienceWorld and SynthProc show cache-read fractions of .93–.94 and .74–.90, with Skill Block reductions of −53.3% and −73.1%, respectively.Hybrid reductions are −37.7% and −67.0%, while reference reductions are −20.2% and −14.9%.
  • C.1 Cache fractions, means, and output sensitivity: At λ=4 and λ=8, Skill Block reduces total_λ by −46.8% and −37.2% for Science, and −53.7% and −37.2% for Synth.These totals use raw input for single-turn rows and effective input for multi-turn rows.
  • C.2 Discount and cache-write sensitivity: Cache-policy sensitivity makes SearchQA Skill Block and reference cross the full arm near d = .354 and d = .440, respectively.Raw input remains the primary metric for single-turn benchmarks, while ScienceWorld and SynthProc Skill Block beat full by approximately −60% and −54%.
  • C.3 Statistical details: ScienceWorld’s three outcome comparisons each have Holm-adjusted p=.116, supporting “no detected difference” but not equivalence or non-inferiority.The reported bootstrap intervals are 95% paired-resampling intervals and are not simultaneous family-wise token intervals.
  • D.1 SynthProc configured as gpt-5.6: At λ=4, SynthProc Skill Block and hybrid are −60.9% and −60.4% versus full; at λ=8, they are −56.1% and −57.2%.The medium-effort supplementary run gives hybrid −68.4%, Skill Block −66.4%, and reference −8.9%.
  • D.1 SynthProc configured as gpt-5.6: SynthProc evaluation requires evaluator isolation, exposing only arm-specific skills, registered tool results, and current observations while withholding synthetic ground truth files and trajectories.The model must not access ops.json, tasks.jsonl, tiers.json, the unrestricted full skill, or historical result trajectories.
  • D.1 SynthProc configured as gpt-5.6: SynthProc paired median effective deltas are −10,013 for Skill Block, −10,329 for hybrid, and −3,279 for reference.Skill Block and hybrid are cheaper in 40/40 pairs, while reference is cheaper in 35/40.
  • D.2 Real ScienceWorld configured as gpt-5.6: Real ScienceWorld mean effective deltas are −58.5% for Skill Block, −61.1% for hybrid, and −49.1% for reference, with outcome tests remaining non-significant after Holm adjustment.Skill Block, hybrid, and reference are cheaper on 89/90, 89/90, and 88/90 paired comparisons, respectively.

Appendix E: Extended Limitations and Design Details

The appendix qualifies the study’s design, attribution, and scope, emphasizing that loading guidance, caching behavior, endpoint drift, and block construction constrain interpretation. It also outlines benchmark-specific limitations and a deployment checklist for content-parity comparisons.

  • Prompt scaffolding and guidance: Aggressive SearchQA loading guidance yields +48.4% for pure Skill Block, while pilot guidance used alternative, less frequent loading and was not confirmatory evidence.These guidance changes are part of deployment rather than removable nuisance text.
  • Schema cache attribution: Observed cache patterns are consistent with provider prefix caching, but logs cannot attribute hits to schemas versus conversation content.Different provider billing or cache-write policies may shift the boundary between Skill Block and hybrid.
  • Size gate and post-hoc model: The approximately 2K pure-Skill-Block size gate reflects this endpoint’s observed overhead, does not apply to hybrid, and was not evaluated as a break-even curve.The associated outcome model organizes observed loading behavior rather than preregistered predictions.
  • Temporal and endpoint drift: Concurrent, nonrandomized execution and later historical runs limit separation of endpoint drift from smaller outcome differences and discourage stable-rank claims.The passage states that large token gaps are unlikely to be explained entirely by drift.
  • Other unresolved scope: Blocks are mostly hand authored, multilingual and paraphrase routing were not directly tested, and wall-clock latency, provider invoices, and a separate historical reasoning-token field were not measured.The appendix also identifies cross-vendor replication, learned partitioning, and routing controls as unresolved scope.
  • Availability versus necessity: The four primary SpreadsheetBench arms preserve all 14 blocks, whereas historical static original8 removes six without establishing shifted-coverage equivalence or isolated retrieval causality.Dynamic fetches demonstrate selection occurred, not that every fetched block was causally necessary.
Loading 2608.14943v1…