PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

Shuyang Zhang

arXiv:2609.33153v2cs.SEcs.AI

TL;DR

Tool and skill evaluations report gains from different comparisons, leaving unclear which conclusions their designs support. This review characterizes evaluation contrasts and uses analytic decompositions and counterexamples to examine their interpretation. It shows that paired runs alone do not identify an invocation effect under trigger conditioning, and distinguishes that question from total deployment effects and budget-constrained efficiency.

  • Problem

    Evaluations of tools and skills use different treatment contrasts, populations, and trigger-conditioned comparisons, complicating interpretation of reported gains.

    Source: §1 · p. 2

  • Method

    The review characterizes evaluation designs along six axes and analyzes recurring interpretations with analytic decompositions and counterexamples.

    Source: §1 · p. 2

  • Results

    Pairing runs on the same task does not by itself identify an invocation effect when evaluation conditions on a trigger within the treated run.

    Source: §1 · p. 2, §3.2 · p. 6, §3.5 · p. 11

  • Takeaways & Limitations

    Total effects of deploying a module and efficiency under a common budget answer different questions and should be reported as distinct comparisons.

    Source: §1 · p. 2

    The review’s mechanism evidence for presence-only flips comes from one run per task, with labels assigned by one author and no metadata-only arm.

    Source: §3.6 · p. 12

Abstract

from arXiv · show

Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative review examines what these evaluations estimate and which conclusions their designs support. The review checks the roles of one hundred cited papers and extracts focal evaluation designs in detail from thirty-five studies. Targeted readings of thirty-five additional published or accepted studies broaden coverage of tool creation, memory, interactive benchmarks, reliability, and risk. Designs are characterized by treatment contrast, target population, outcome, budget constraint, summary measure, and identification assumptions. Analytic decompositions and counterexamples show that pairing runs on the same task does not itself identify an invocation effect when evaluation conditions on a trigger within the treated run. Paired gain and regression counts describe discordance under the coupling protocol rather than the share of tasks whose expected outcomes worsen. Total effects of deploying a module answer a different question from efficiency under a common budget. Comparisons across studies distinguish curated skill provision from retriever replacement, task populations from triggered subsets, and preparation costs from marginal usage costs. Publication status and reading depth are recorded. The review provides a methodological synthesis and a reporting checklist to help align claims about tools and skills with the comparisons their evaluation designs support.

1 Introduction

Tool and skill evaluations compare different system changes and target populations, so their reported gains do not answer one common question. This review separates measurement levels from causal paths and analyzes how design choices shape the conclusions evaluations support.

1 Introduction

  • Tool use connects LLMs to APIs and environments, while reusable skills add routines or verified programs; selection occurs at retrieval, loading, and use.These decisions change different parts of the system and require different evaluation contrasts.Source: §1 · p. 1
  • ToolRet evaluates retrieval and downstream success after retriever replacement, while AWM and ASI report gains from deploying adaptation packages.Other studies compare accuracy with cost and resampling baselines, curated skill bundles with no skills, or paired gains with regressions [13] [14].Source: §1 · pp. 1–2
  • Trigger-conditioned contrasts and paired outcome flips require careful interpretation; paired counts do not directly establish how often tasks’ expected outcomes worsen.The review focuses on how within-run trigger conditioning interacts with stochastic-run coupling [20].Source: §1 · p. 2
  • The review separates five measurement levels from causal paths and characterizes evaluation designs along six axes using potential-outcome notation.It also analyzes recurring interpretations with analytic counterexamples and provides a reporting checklist linked to applicable designs.Source: §1 · p. 2

2 Scope and Review Method

This critical narrative review purposively selects literature to examine evaluation designs for tool and skill components. It records reading depth and publication status but does not pool effects or grade study quality.

2 Scope and Review Method

  • Selection was purposive, focusing on comparisons that differ in treatment, population, pairing, or budget, and included relevant evaluation-protocol studies.The search covered arXiv, the ACL Anthology, and conference proceedings, with backward and forward citation tracking [11].Source: §2 · p. 3
  • The review checks the roles of 100 papers, fully extracts focal designs from 35 studies, and conducts targeted readings of 35 additional papers.The detailed and targeted groups are disjoint; another 30 papers, a book, and a guideline provide methodological or background support.Source: §2 · p. 3
  • Detailed extraction records focal arms, units, populations, inclusion rules, runs and coupling, conditioning events, budgets, endpoints, versions, and supporting locations.Extraction covers cited comparisons rather than every experiment in each paper; unreported details are identified relative to cited sections.Source: §2 · p. 3
  • The topical map documents selected literature rather than an exhaustive search or quality ranking, and study quality is not graded.Publication status is reported separately from design validity; the thirteen frontier cases illustrate protocols, not prevalence or general effect sizes.Source: §2 · p. 4

3 A Framework for Tool and Skill Evaluation

The framework distinguishes exposure, retrieval, activation, use, and outcome while treating their sequence as a typical workflow rather than a complete causal graph. It emphasizes precise event definitions because studies condition on different events.

3.1 Measurement Levels and Causal Paths

  • The framework distinguishes five levels: exposure, retrieval, activation, use, and outcome.Exposure concerns what candidates are visible; retrieval ranks candidates; activation loads or executes them; use concerns whether and how the agent follows them; outcomes include success, cost, latency, and risk.Source: §3.1 · p. 5
  • The typical sequence is not a complete causal graph: exposure can affect outcomes directly, actions feed back into later states, and the sequence can recur.Skill names or descriptions can change behavior without invocation, and larger context can matter even when selection is correct [13] [45].Source: §3.1 · p. 5, Figure 1 · p. 6
  • Event definitions distinguish surfaced candidates, retrieval calls and returns, loaded skills, executed tools, and followed content.A request returning no candidates differs from one returning candidates, so the trigger rate and triggered group depend on the event chosen.Source: §3.1 · p. 5
  • Executable skills can span several primitive steps, making action granularity and internal execution cost relevant when calls count as single high-level steps.This connects bundled skills to temporal abstraction in the options framework.Source: §3.1 · pp. 5–6

3.2 Notation, Units, and Coupling

The review defines agent outcomes and run coupling, then organizes evaluation designs by six axes. It distinguishes descriptive quantities from total effects and explains how module availability, component swaps, and budget constraints change the question being answered.

3.2 Notation, Units, and Coupling

  • Agent runs are stochastic at a fixed task, so comparisons must specify the unit, arm coupling, and runs per task and arm.A shared seed defines a joint sampling protocol but does not guarantee similar trajectories; run counts determine how precisely task-level expectations can be estimated.Source: §3.2 · p. 6

3.3 Six Axes for Describing a Design

  • Evaluation designs are characterized by treatment contrast, target population, outcome, budget constraint, summary measure, and identification assumptions.Populations may include all tasks, pre-treatment subgroups, post-treatment trigger subsets, or trajectory states.Source: §3.3 · p. 7

3.4 Commonly Reported Quantities and Their Limits

  • Retriever metrics and retrieved-versus-skipped contrasts are descriptive, while same-task comparisons can identify total effects of module availability or component replacement.A component-swap effect concerns replacement of one component by another, not the effect of skills relative to none.Source: §3.4 · pp. 7–8
  • A module’s overhead belongs to its total deployment effect, whereas choosing under a spending cap requires a separately specified budget-constrained comparison.The comparison needs an ex-ante cap and an allocation rule for each arm.Source: §3.4 · p. 8

3.5 Trigger-Conditioned Paired Contrasts

Pairing triggered module runs with controls removes between-task difficulty differences but does not by itself identify an invocation effect. The paired contrast can reflect task effects and selection associated with triggering and the coupling protocol; availability effects and local invocation effects are distinct targets.

3.5 Trigger-Conditioned Paired Contrasts

  • Pairing each task’s triggered module run with a module-unavailable run removes between-task difficulty differences, but the restricted contrast is not automatically a causal invocation effect.Conditioning on a trigger selects on a post-treatment event, and the paired result remains a protocol-conditional diagnostic rather than a model-level measure of skill use.Source: §3.5 · pp. 8, 11
  • The paired statistic combines trigger-probability-weighted task effects with treated-run and coupling-dependent control-run selection terms.Under independent control runs, the control-run selection term vanishes; dependent coupling can make it nonzero and can partly or fully offset treated-run selection.Source: §3.5 · p. 10
  • With identical tasks and a true zero effect, trigger-on-success yields an expected paired contrast of 0.5 with independent controls but 0 under a shared coin.The example demonstrates that pairing does not remove within-task selection and that bias depends on coupling; it does not establish the bias under any particular shared-seed protocol.Source: §3.5 · p. 10
  • Skill Following’s shared task prompt, generation seed, and decoding define dependent coupling, but the link between retrieval randomness and control-run success is unspecified.The review therefore treats both selection terms and their difference as unknown for this protocol.Source: §3.5 · p. 10
  • Availability effects among runs that would trigger and local effects of loading at a trigger state are different targets, and neither implies the other.The local invocation target holds metadata and remaining budget fixed and specifies a continuation policy; the availability target includes all paths through which module availability acts.Source: §3.5 · p. 11

3.6 Linking Total and Trigger-Conditioned Contrasts

The overall paired mean decomposes into trigger-conditioned and skipped contrasts, weighted by the trigger rate. This identity is descriptive rather than causal, and a skipped contrast alone cannot establish a presence effect.

3.6 Linking Total and Trigger-Conditioned Contrasts

  • The overall paired mean equals the trigger rate times the triggered paired contrast plus the remaining share times the skipped contrast.The identity holds for any protocol and coupling, without establishing a causal interpretation for either component.Source: §3.6 · p. 11
  • A complete report gives the trigger rate and both conditioned contrasts, or the trigger rate, overall paired mean, and triggered contrast.At trigger rates of zero or one, the contrast for the empty subgroup is undefined.Source: §3.6 · p. 11
  • A nonzero skipped contrast is consistent with a presence effect but does not demonstrate one because selection terms also contribute.Attributing the contrast to metadata requires a metadata-only arm; Regression Tax reports trajectory evidence of presence-only flips but uses one run per task and author-assigned mechanism labels, without that arm.Source: §3.6 · p. 12

3.7 Discordance and Degradation

Paired gains and regressions describe discordance under a specified run protocol, not the share of tasks whose expected outcomes worsen. Estimating that degradation share requires repeated-run evidence and assumptions beyond paired counts.

3.7 Discordance and Degradation

  • Marginal arm success rates do not determine paired regressions: at 0.5 success per arm, regression probability ranges from 0 to 0.5 across the stated couplings.It is 0.25 under independent runs; repeated runs estimate the protocol-specific rate more precisely, but neither repetition nor a same-configuration noise floor identifies causally harmed tasks.Source: §3.7 · p. 12
  • Paired gain and regression counts describe protocol-specific discordance, while task-level degradation is the share of tasks whose expected outcome falls by more than ε.For binary success with one run per arm per task, the paired difference in success rates equals the gain share minus the regression share; the regression share alone is not the harm share.Source: §3.7 · p. 12
  • Degradation-share estimation error depends on runs per task and the number of tasks near the margin, so no fixed small run count makes a naive estimate reliable.The review found no core study reporting δ_ε or either proposed interval-based share.Source: §3.7 · p. 13
  • With both arms succeeding at probability 0.5 on every task, three independent runs per arm yield a plug-in degradation estimate centered near 0.34 although the true δ0 is zero.Adding tasks makes this biased estimate more stable rather than removing the bias.Source: §3.7 · p. 13
  • Simultaneous task-level intervals can bound the evaluated-benchmark degradation share through confidently and possibly degraded task counts.Extending the bound to a wider population additionally requires a task sampling design and population-level interval; alternatively, a hierarchical model requires stated assumptions.Source: §3.7 · p. 13

3.8 State-Level Intervention Value

State-level intervention value compares taking an action with not taking it at a particular state while holding the subsequent policy and budget fixed. Identifying that value from logs requires assumptions about actions and the later trajectory; nearby skill and execution gates target different quantities.

3.8 State-Level Intervention Value

  • The intervention value vφ compares outcome and total subsequent cost when loading a candidate skill versus not loading it, with policy φ fixed for the rest of the trajectory.The cost difference covers total subsequent cost, not only the cost of loading the skill.Source: §3.8 · p. 13
  • Logged-run estimates equal vφ only when both arms follow φ afterward and consistency, current-decision exchangeability, and positivity hold.If logged runs continue under another policy, raw means estimate that continuation; off-policy estimation additionally requires sufficient recorded history, action support, and continuation-policy conditions.Source: §3.8 · p. 14
  • SkillApt is the closest core-study approximation, using matched WITH and WITHOUT executions, but its confirmatory value is task-conditional and based on 111 of 113 complete states.Correctness is primary, costs break ties, and infrastructure-failure pairs are censored.Source: §3.8 · p. 14
  • Confidence-gated retrieval on fixed paths swaps confidence-to-action controllers; it does not estimate the incremental value of another retrieval.Calibration, controller comparison on fixed paths, and value-of-information estimation are distinct targets.Source: §3.8 · p. 14
  • RADEG gates whether to execute the downstream agent on a query–bundle pair, not whether to load a skill within the same run.Its prediction of non-zero verifier reward is distinct from expected reward and execution cost; its logged-rollout evaluation requires support for both decisions and a fixed execution mechanism.Source: §3.8 · pp. 14–15
  • Accurate local gate estimates do not guarantee a beneficial induced policy, so the gate’s total effect or policy value must also be evaluated.This distinguishes local state- or query-level estimates from the overall policy outcome.Source: §3.8 · p. 15

4 Evidence Synthesis

This evidence synthesis examines how evaluation designs for tools and skills define their interventions, target populations, outcomes, and execution protocols. It distinguishes evidence about policy effects, repeated-run reliability, safety in constructed scenarios, and deployment risk, aligning conclusions with the specific components or deployment packages measured.

4.1 Evaluation Settings and Target Populations

  • Benchmark scores measure different populations and endpoints, so recorded-state action prediction, tool-call correctness, and interactive task completion are not interchangeable outcomes.For AWM, Mind2Web’s recorded-state prediction and WebArena’s executable task completion are complementary findings, not estimates of the same endpoint. Aggregate scores across benchmarks also depend on task mixtures and score definitions.Source: §4.1 · p. 17

4.2 Exposure and Candidate Construction

  • Retrieval comparisons support specific component choices, but ranking or reference-set coverage alone does not establish execution success or coverage of all valid solutions.Re-Invoke’s downstream comparison holds its executor fixed but compares retrievers, while the State-Path menu study supports menu replacement only in its tested setting; neither establishes a universal surrogate for task success.Source: §4.2 · pp. 18–19

4.3 From Retrieval Gains to Outcome Gains

  • SkillsBench’s curated, task-specific bundles raised macro pass rate from 33.9% to 50.5% across 18 configurations [14], while self-generated skills fell below the no-skills baseline in three tested configurations.Source: §4.3 · pp. 19–20

4.4 Activation and Actual Use

  • SkillApt matched BM25 Top-1 observed accuracy at 0.838 while reducing activation from 100% to 31.5%, illustrating that retrieval relevance and the value of loading are distinct decisions.Skill Following’s triggered, paired contrast is protocol-conditional, not the all-task availability effect; paired gain and loss counts likewise do not estimate the share of tasks with lower expected success. Module removal, reference-procedure conformity, and observed call frequency are also distinct from outcome change under intervention.Source: §4.4 · pp. 20–22

4.5 Library Scale, Evolution, and Cost

  • Expanding initially helpful skill sets to 202 skills lowered pooled pass rate by 21 percentage points, a conditional case study rather than a typical scaling effect.The study selected 38 task–model pairs where every authored skill improved pass rate by at least 4 percentage points; its decomposition depends on stated assumptions. Budget comparisons also require matched actors, environments, allocation rules, and resource budgets.Source: §4.5 · pp. 22, 24
  • Observed cost–success profiles do not establish equal-budget efficiency, and preparation costs must be distinguished from marginal use costs.LATM amortizes expensive tool construction across task instances, while the WebArena comparison of augmentation methods with a strengthened control changes both actor horizon and accessibility-tree pruning and does not exactly match token budgets.Source: §4.5 · pp. 23–24

4.6 Validity of Evaluation Protocols

  • StableToolBench’s scores concern 765 tasks selected as solvable in a stabilized environment, while repeated grading does not measure run-to-run execution variation.In a replay validation study, swaps rewrote 61–94% of post-fork actions, only 3% of replayed states remained valid, and replay mispredicted every success-relevant outcome; changed components therefore need executed continuations unless replay is validated for them.Source: §4.6 · p. 24
  • A critic with offline AUROC 0.94 caused a 26-percentage-point collapse on one model and almost no change on another, showing local accuracy does not establish intervention benefit.The authors attribute this to a disruption–recovery trade-off and propose a 50-task pilot; activation-gate accuracy likewise does not determine the policy’s total effect [48].Source: §4.6 · pp. 24–25
  • Over 90% of control forks diverged under one quantization configuration, and run-to-run variance changed reported conclusions [47] [12].Random seeds and implementation choices can also change performance; aggregate uncertainty intervals are recommended, but do not identify a task-level degradation share.Source: §4.6 · p. 25
  • Pass-hat-k measures the probability all k independent trials succeed, whereas pass-at-k measures the probability at least one succeeds; neither estimates paired harm.Reliability comparisons should retain the same task population and user-simulation protocol rather than substitute paired gain or regression counts for repeated-success outcomes.Source: §4.6 · p. 25
  • ToolEmu and ToolSandbox assess failures within emulated or authored scenarios, not their frequency in ordinary deployment.Safety conclusions should name the event and scenario-selection process rather than equate stress-test failure rates with operational risk.Source: §4.6 · p. 25
  • Adversarial benchmarks measure distinct outcomes: following injected instructions, preserving user-task utility, and completing harmful workflows.InjecAgent distinguishes ASR-valid from ASR-all; AgentDojo evaluates attacks, defenses, and utility; AgentHarm scores malicious-task progress and refusal. Rates depend on attack distribution, injection opportunity, and scoring rules, not ordinary deployment harm prevalence.Source: §4.6 · pp. 25–26
  • Off-policy skill-gate estimates require target-policy actions to be supported by logged actions, plus consistency, propensity, and other assumptions [67, 68, 102].Logs must cover both loading and abstaining, and specify the continuation policy; accurately predicted logged outcomes do not identify unsupported alternatives.Source: §4.6 · p. 26
  • Progress checkpoints, reference-tool coverage, repeated-success reliability, and final success are different endpoints and should not be treated as interchangeable measures of capability.AgentBoard adds state-matching or annotated-subgoal progress measures, while ASI uses intermediate checkpoints in scaled-up activity experiments and task success in ordinary WebArena evaluation.Source: §4.6 · p. 26

5 Discussion

The discussion distinguishes evaluation questions that are often conflated and recommends reporting the design details needed to interpret them. It also identifies open problems in evaluating invocation, counterfactuals, evolving libraries, and policy effects, while noting limits of the review itself.

5.1 Cross-cutting Lessons

  • Pairing on tasks does not remove within-task selection in trigger-conditioned contrasts, and gain or regression counts describe protocol-specific discordance rather than tasks harmed.Invocation claims require intervening on the load decision; claims about expected degradation require repeated per-task evidence with uncertainty bounds or a stated model.Source: §5.1 · p. 26
  • Total effects answer whether to deploy a module as specified, whereas budget-constrained comparisons or cost–success frontiers address configuration choice and resource efficiency.Replay and counterfactual methods impose different constraints; their validity for deployment claims depends on which actions and responses may change after intervention.Source: §5.1 · pp. 26–27

5.2 Reporting Checklist

  • Table 6 presents a reporting checklist whose items specify the designs they apply to, and most can be reported from runs authors already perform.The examples illustrate interpretation rather than score studies or validate the checklist.Source: §5.2 · p. 27, Table 6 · p. 28
  • Skill Following’s triggered-subset contrast supports a protocol-specific diagnostic, not a local invocation claim without a load intervention and specified metadata, budget, and continuation policy.The source describes its retrieval-invoked contrast as protocol-conditional [11].Source: §5.2 · p. 27
  • Regression Tax’s one run per condition and task describes execution flips, but does not estimate run-to-run variance or the share of tasks whose expected success decreases [13].That stronger claim would require repeated per-task evidence with simultaneous uncertainty bounds or a stated model.Source: §5.2 · p. 27
  • The web-agent budget study supports comparison with its longer-horizon control package, but an exact equal-budget claim needs a common ex-ante cap and allocation rules for every arm [12].The review preserves the study’s qualification of its budget control.Source: §5.2 · p. 27

5.3 Open Problems

  • For run-level triggers, identifying stratum or local invocation effects while modeling trigger variation and arm coupling remains an open problem; cheaper valid counterfactuals and benchmarks with built-in controls are also needed.Proposed benchmark controls include metadata-only, no-skill, and full-access arms, plus branching from saved states under fixed continuation policies.Source: §5.3 · pp. 27, 29

5.4 Limitations of This Review

  • The detailed studies were selected for methodological relevance rather than publication-frequency representativeness, and the review did not reproduce experiments or infer replication from publication.Evolving libraries raise a separate open question: evaluating an admission policy without rerunning the full evolution.Source: §5.4 · p. 29, §5.3 · p. 29

6 Conclusion

Tool and skill evaluations use different contrasts, populations, and protocols, so reported improvements do not all estimate the same quantity. The review distinguishes what these designs support and argues that precise reporting can help readers compare apparently conflicting results.

6 Conclusion

  • Tool and skill evaluations mix retrieval metrics, component swaps, retrieved-versus-skipped and trigger-conditioned contrasts, discordance counts, and budget-constrained comparisons.These designs address different questions, and some describe a protocol without identifying a mechanism.Source: §6 · p. 29
  • Task pairing does not make a trigger-conditioned contrast an invocation effect, regression counts describe discordance rather than harm, and an unmatched-budget comparison estimates a total effect.The absence of budget matching does not make that total effect invalid.Source: §6 · p. 29
  • Stating each design’s contrast, population, unit and coupling, budget, summary, and identification assumptions helps readers compare results that currently appear to conflict.The review concludes that retrieval evaluation requires stating precisely what each design shows.Source: §6 · pp. 29–30

A Study Evidence and Reading Register

The appendix separates detailed extraction of focal evaluation designs from targeted readings that broaden coverage; neither group is a quality ranking.

A Study Evidence and Reading Register

  • The appendix lists thirty-five focal studies in Table 7 and thirty-five additional targeted readings in Table 8, with neither grouping serving as a quality ranking.Section 2 describes the roles of the other thirty papers and two non-paper sources.Source: A Study Evidence and Reading Register · p. 30

A.1 Detailed Study-Level Evidence

This section inventories study-level evaluation designs for tools, memory, retrieval, and safety, recording their contrasts, populations, outcomes, budgets, and identification conditions. It highlights how task selection and evaluation protocols shape reported skill-library effects, while unspecified repeat or seed coupling and varied resource constraints limit common-budget comparisons.

Detailed study-level evidence

  • Table 7 records focal comparisons by arms, population, repetition, budget, endpoint, and evidence locations, distinguishing independent executions from adaptive attempts, folds, candidate samples, raters, and bootstrap resamples.Publication status and reading depth are also recorded; “not specified” means the cited locations do not state a design feature, not that it was absent.Source: §A.1 · p. 30, Table 7 · p. 30
  • Other focal designs varied in population and constraint: StableToolBench used 765 solvable tasks with three evaluations, while Failure prevention [48] used intervention caps without a matched total-token or cost budget.StableToolBench’s main comparison used one execution per model and distinguished inference from evaluator variations; Failure prevention reported success, recovery, and disruption counts across HotPotQA, GAIA, and ALFWorld.Source: §A.1 · pp. 31, 33
  • 0.21 (21 percentage points) was the pooled pass-rate drop at 202 skills, while the largest single-model drop was 0.26 for Haiku 4.5 in Skill Shadowing [45].The examined SkillsBench pairs were retained only when every skill in the authored bundle raised pass rate by at least 4 percentage points over no skills; the version does not state whether filtering and estimation used separate runs. The cited version reports 2,545 trajectories, but Table 2 condition counts sum to 2,576.Source: §A.1 · p. 32
  • SRA-Bench’s frozen confirmatory split contained 113 states, with 111 complete after two no-skill timeouts were censored; the WITH/WITHOUT comparison used one execution per pair and paired-bootstrap uncertainty.Correctness was primary and cost a tie-break; separate gate evaluation used fixed call budgets and predicted non-zero verifier reward from query–bundle features.Source: §A.1 · p. 32
  • Replay Gap [47] compared model swaps with same-model control forks at controlled points in live SWE-bench trajectories; its roughly 900 branch continuations were not independent tasks.The study used six run pairs, each a 30-instance sweep in one swap direction, with temperature 0, action edit distance and resolution outcomes, and step budgets.Source: §A.1 · p. 32

A.1 Detailed Study-Level Evidence

  • GPT-4-made tools were compared with CoT and direct few-shot baselines across six reasoning tasks, with five maker trials but unspecified downstream repeats and test-set coupling.The study measured accuracy and cost, allowed three proposal/verification retries and one LLM use call, and amortized construction costs.Source: §A.1 · p. 34
  • Reflexion [27] compared ReAct with reflective memory against restart without reflection, while experience-retrieval experiments compared insights and retrieved trajectories with ReAct/Act.Reflexion used 134 ALFWorld environments and adaptive trials; retrieval experiments used four-fold validation across HotpotQA, ALFWorld, WebShop, and FEVER transfer, without matched preparation-plus-evaluation budgets.Source: §A.1 · p. 35
  • Voyager tested library ablations and transfer with or without a library in Minecraft, using three trials and iteration limits rather than a token cap.Transfer involved four unseen tasks in a new world with inventory reset; outcomes included item and tech-tree progress and task completion.Source: §A.1 · p. 35
  • Generative Agents compared full memory with three nested access ablations using human ratings after two simulated days, not repeated simulations.One hundred within-subject raters supplied believability ranks and TrueSkill scores; histories were shared and not regenerated by ablated agents, and no common inference budget was stated.Source: §A.1 · p. 35
  • LLMCompiler [96] compared parallel dependency execution with ReAct variants and parallel calling on HotpotQA, movie recommendation, and ParallelQA, measuring accuracy, latency, and token cost.Three-run mean accuracy was reported in Appendix D, but seed coupling and a common spending cap were not stated; latency controls differed by scenario.Source: §A.1 · p. 35
  • Simulated-user evaluations tested function-calling and ReAct/Act strategies on authored retail and airline tasks, while separate adversarial-emulator tests used 144 curated risk cases.The former measured database/response success and repeated-success reliability under a 30-action limit; the latter assessed safety, helpfulness, and failures without a common resource cap.Source: §A.1 · p. 36
  • InjecAgent [99] assessed base and enhanced attacks across 1,054 cases per setting, while AgentDojo [100] measured benign utility and targeted attack success across 97 tasks and 629 task/attack-target cases.InjecAgent’s attack measure covers the next harmful call or extraction sequence, not a full workflow; AgentDojo reports confidence intervals, but cited setups do not specify execution repeats or a common cross-agent cap, and aggregation weights matter.Source: §A.1 · p. 36
  • Re-Invoke [21] compared its retriever with ToolLLM and supplied reference tools on six ToolBench subsets; Better Turns [51] [A] evaluated SFT using held-out conversations and workflow outcomes.Re-Invoke’s reproduced pass rates lack a stated repetition count and include offline indexing plus an extra online LLM call under limited execution budgets. Better Turns used 84 held-out conversations, 542 next-action examples, and 77 tool-requiring workflows, with gold-history scoring and model-history replay.Source: §A.1 · pp. 36–37

A.2 Targeted Readings by Topic

Table 8 broadens the review with thirty-five targeted readings that clarify systems, populations, endpoints, and operating assumptions, without presenting complete design extractions. Across these examples, benchmark outcomes and tool or skill comparisons depend on task-specific measurement rules, intervention packages, and resource protocols.

Targeted reading register

  • Table 8 records thirty-five further targeted readings that support definitions of systems, populations, endpoints, and assumptions, rather than complete design extractions.The thirty-five studies in Table 7 are focal design extractions; the two groups are disjoint and neither is a quality ranking.Source: §A.2 · p. 37, Table 8 · p. 37

Interactive benchmarks

  • Task endpoints are not interchangeable: WebShop separates graded reward from full instruction satisfaction, while ScienceWorld scores subgoal progress rather than binary success.ALFWorld also links textual policies to embodied actions, with scene and domain shifts affecting transfer.Source: §A.2 · p. 38
  • GAIA scores normalized final-answer matching, so correct answers do not verify a particular tool invocation or web-state change; AgentGym comparisons depend on environment weights and endpoint interpretation.Visually grounded web tasks use task-specific binary validators, sometimes learned, so a common success label need not represent the same measurement rule.Source: §A.2 · pp. 38–39
  • Fixed application versions and seeded task parameters define the sampled population, while action limits and termination rules remain part of the resource protocol.Workplace task levels also change information availability, so performance gaps cannot isolate retrieval alone.Source: §A.2 · p. 39

Tool construction and training

  • ToolkenGPT changes calling behavior with learned tool-token embeddings, while CRAFT and CREATOR evaluate packages involving creation, verification, retrieval, or rectification.CREATOR combines creation, decision, execution, and rectification rather than isolating a single intervention.Source: §A.2 · p. 39

Tool creation and evaluation

  • AnyTool’s success concerns manually retained solvable queries and its chosen evaluator; ToolGen’s trained tool-token intervention is broader than external retriever replacement.ToolACE validation ablations also change retained dataset size, and synthetic-data checks do not directly measure live API execution.Source: §A.2 · pp. 39–40

Planning, revision, and search

  • Search and revision comparisons depend on inference allocation, stopping rules, search budgets, and environment-state restoration, so full-system results do not isolate individual mechanisms.Bounded tree search success on selected reasoning problems does not directly measure a retrieved skill component.Source: §A.2 · pp. 40–41

Memory, reliability, and risk

  • Memory benchmarks measure answer quality rather than autonomous tool-trajectory success, while AgentHarm scores proxy harmful-task progress and refusal without real-world side effects.ToolQA error analysis separates argument, source-selection, and hallucinated-observation errors but does not establish causal stage contributions or rule out memorization for every future model.Source: §A.2 · pp. 38, 41

Related papers

Loading 2609.33153v2…