PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

Jinlin Liu, Wei Chen, Xiaojin Zhang

arXiv:2610.18522v1cs.AI

TL;DR

Reusable skills require agents to select relevant instructions from growing metadata libraries, but the scalability of this implicit routing remains uncertain. The paper compares routing and preselection strategies and finds that irrelevant descriptions and choosing skills during execution are associated with poorer selection. Masking metadata partly restores loading, while fixed preselection can omit skills needed later.

  • Problem

    As skill libraries grow, it is unclear whether agents can reliably select relevant skills from metadata while executing tasks.

    Source: §1 · p. 1

  • Method

    The paper compares implicit execution with metadata masking, routing-only preselection, and Plan-first selection while adding irrelevant skills to a fixed task library.

    Source: §4.2 · pp. 6–7

  • Results

    Growing libraries make agents load fewer or no skills, and with metadata intact Plan-first recalls 39.9% versus 70.6% for isolated preselection.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Descriptions act as semantic-matching inputs during routing-only selection but as conditional sub-instructions during execution, a distinction that helps explain more reliable isolated selection.

    Source: §4.2 · p. 7

    Fixed preselection can omit skills needed later.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.

1 Introduction

The paper tests whether implicit routing can still find relevant skills as irrelevant skill descriptions accumulate. It finds that growing libraries suppress loading, while task success declines less sharply and explicit preselection is evaluated as a mitigation.

1 Introduction

  • Implicit routing asks agents to choose whether and which skills to load from names and descriptions while planning and solving the task.This progressive-disclosure design keeps long skill files out of context but requires selection amid execution.Source: §1 · p. 1
  • The evaluation holds SkillsBench task instructions, workspaces, verifiers, and relevant skills fixed while adding irrelevant skills.This isolates the effect of library growth on skill loading.Source: §1 · pp. 1–2
  • With 1,000 irrelevant skills, implicit loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%.Explicit selection from the same library retains 73.9% selected-skill recall, and loading non-golden alternatives is too rare to explain the collapse.Source: §1 · p. 2
  • End-to-end Pass@3 falls less sharply, from 51.6% to 38.6%, because strong agents can sometimes solve around missing skills.Task success alone can therefore make a failing skill system look healthy.Source: §1 · p. 2
  • The paper identifies routing collapse as agents loading fewer or no skills, rather than substituting other skills for golden ones.It also compares LLM, retrieval, and hybrid preselection on libraries with up to 10,000 added skills, including token, coverage, and latency costs.Source: §1 · p. 2
  • Irrelevant descriptions suppress skill loading, while choosing skills during task execution makes selection less reliable.The authors characterize these effects as semantic and instruction interference beyond context growth alone.Source: §1 · p. 2

2 A Controlled Test for Skill Routing

The study fixes each task’s instructions, workspace, verifier, and golden skills while varying irrelevant skills. It compares implicit routing with explicit preselection and defines metrics for skill loading and task success.

2.1 Task setup

  • Each task is represented by its instruction, workspace, executable verifier, and golden skills; success requires the verifier to pass.The installed library combines the task’s golden skills with independently sampled irrelevant skills, and only the number of irrelevant skills varies.Source: §2.1 · p. 2

2.2 Routing strategies

  • Implicit routing lets the executor decide during execution whether to open skills, while explicit routing preselects a compact subset for the executor.Preselection can use an LLM, embedding retrieval, or retrieval followed by an LLM choice; routing-only diagnosis tests recognizability, whereas preselection-plus-execution tests practical benefit.Source: §2.2 · p. 2

2.3 Metrics

  • Pass@3 measures task success, while loaded-skill recall, zero-load rate, and selected-skill recall measure whether golden skills are opened or preselected.Loaded-skill recall is the fraction of golden skills opened during execution; selected-skill recall is the fraction included in the preselected set.Source: §2.3 · p. 2

3 Main Findings

Across controlled SkillsBench scaling tests, implicit skill use degrades as irrelevant libraries grow, while explicit preselection reduces scale sensitivity but brings coverage and latency trade-offs.

3.1 Experimental setup

  • The experiments hold SkillsBench instructions, workspaces, verifiers, and golden skills fixed while varying added distractors, using three open-weight executors across 82 tasks.The evaluated library sizes range from 0 to 10,000 added skills, and each preselector exposes at most five skills.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2.1 Implicit loading degrades before task success

  • At 100 irrelevant skills, loaded-skill recall falls by 24.6, 15.6, and 20.7 points for GLM-5, Kimi K2.5, and MiniMax M2.5, while task success changes less.At that scale, GLM-5 Pass@3 moves from 57.3% to 56.1% and Kimi K2.5 from 52.4% to 50.6%; strong executors can solve around missing skills.Source: §3.2.1 · p. 3, Figure 3 · p. 4

3.2.2 Library growth suppresses skill use

  • From zero to 1,000 distractors, average loaded skills fall from 1.72 to 0.66 and zero-load rate rises from 9.2% to 50.8%.Non-golden loading occurs in only 2.32% of evaluations, and loaded-skill precision remains 88.6%, 87.2%, and 76.4% for the three models at 1,000 irrelevant skills, indicating under-loading rather than substitution.Source: §3.2.2 · pp. 3–4, Table 1 · p. 4

3.2.3 The same skills are recognizable when selected explicitly

  • At 1,000 irrelevant skills, GLM-5 loaded 20.9% of golden skills but selected 82.5%, while Kimi K2.5 loaded 40.5% but selected 80.7%.Explicit selection makes skill recognition the sole objective; the full metadata list often suffices when selection is not combined with task execution.Source: §3.2.3 · p. 4, Table 2 · p. 4

3.3 Analyzing Explicit Routing

  • The study compares simple preselectors rather than proposing a new routing algorithm, measuring their benefits when the executor receives only the selected subset.The diagnosis changes the selector’s objective; preselection-plus-execution also narrows the executor’s library.Source: §3.3 · p. 4

3.3.1 Explicit routing across library sizes

  • At 100 added skills, LLM preselection raises aggregate loaded-skill recall from 53.3% to 73.0% and Pass@3 from 49.4% to 54.9%.At 10,000, hybrid preselection reaches 55.2% recall and 50.4% Pass@3, versus 48.5% and 41.5% for RAG; hybrid also improves selected-skill recall over RAG from 62.0 to 72.2 while raising precision from 30.1% to 52.6%.Source: §3.3.1 · pp. 4–5, Figure 4 · p. 5, Table 4 · p. 5

3.3.2 Token cost of routing

  • At 1,000 irrelevant skills, implicit routing uses 1,952k agent-input tokens, while LLM preselection uses 36k routing-input tokens and 497k execution-input tokens, totaling 547k.RAG is cheapest at 10,000 skills at 469k tokens, but performs worst at that scale; hybrid costs 547k, including 8k for preselection, and generalizes better.Source: §3.3.2 · p. 5, Table 5 · p. 5

3.3.3 Why hybrid preselection helps at 10k

  • At 10,000 distractors, the hybrid multi-query candidate pool raises recall from 88.2 to 91.8 and all-gold coverage from 78.0 to 86.2 versus single-query RAG@100.Fixed preselection can omit skills needed later; from 10 to 1,000 distractors, median routing time rises from 8 to 19 seconds for LLM preselection and from 33 to 41 seconds for hybrid.Source: §3.3.3 · pp. 5–6, Table 6 · p. 6

4 Why Agents Are Lost in Implicit Routing

With distractor slots and approximate context length preserved, meaningful descriptions still suppress golden-skill loading; asking agents to plan skill use improves routing but remains weaker than isolated preselection. The results and traces indicate that descriptions interfere with routing and task execution competes with skill selection.

4.1 Experimental setup

  • The study tests two interventions on a 10-task, 1,000-distractor subset using three open-weight models: masking distractor metadata and prompting agents to plan skill use first.Each intervention ran three times per model; golden-skill metadata remained readable in every variant.Source: §4.1 · p. 6

4.2 Results and discussion

  • 63.9% recall with masking versus 73.5% without extra skills, and a 23.6% versus 62.2% zero-load rate, show that meaningful distractor descriptions—not context length alone—drive the observed interference.Masking retained distractor slots at approximately their original length; the failure appears when distractors contain descriptions resembling plausible actions.Source: §4.2 · p. 6
  • 63.3% recall after masking only descriptions approaches full masking at 63.9%, while masking names or locations alone yields 43.7% or 47.5%.Masking names plus locations reaches only 51.7%; remaining descriptions still suppress golden-skill loading beyond competition among explicit loading targets.Source: §4.2 · pp. 6–7, Table 8 · p. 7
  • 39.9% planned recall versus 70.6% with isolated preselection, and 20.5% versus 54.4% full-skill coverage, show that planning skill use remains weaker than choosing skills alone.The settings expose nearly the same task and metadata context; they differ mainly in whether the system prompt also asks the agent to solve the task.Source: §4.2 · p. 7, Table 9 · p. 7

4.3 Case study

  • Two plan-first traces show plans failing to translate into loads: one agent opened only its first planned skill, while another omitted and never loaded the needed visualization skill.Both runs failed; a prompt nudge does not reliably separate routing from execution, whereas narrowing the visible skill set addresses both forms of interference more directly.Source: §4.3 · p. 7, Figure 5 · p. 7

5 Practical Recommendations

The recommendations favor explicit routing as libraries grow, with the choice of preselection method depending on scale. They also call for monitoring skill coverage, latency, and routing metrics alongside task success.

5 Practical Recommendations

  • Around 100 skills, use explicit preselection; for thousands, hybrid retrieval followed by LLM selection is strongest in the experiments, while full-list LLM preselection becomes expensive or infeasible.The hybrid approach keeps the execution context small, but fixed preselection can omit skills needed later and adds latency.Source: §5 · pp. 7–8
  • Track selected and loaded skills, coverage, and response time alongside task success, since Pass@3 alone can hide routing failures.Prompt reminders to consider skills are a lightweight safeguard, but should complement rather than replace preselection.Source: §5 · pp. 7–8

6 Related Work

Related work studies tool choice, reusable experience, and long-context retrieval, while this paper examines routing to skills in an existing library. Skills differ from tools in how their metadata and full instructions enter the agent’s context.

6 Related Work

  • Tool-selection research addresses choosing and invoking APIs, while skill routing requires deciding from compact metadata whether to load fuller instructions or resources.The cited work spans tool use, API selection, and filtering or merging ambiguous tools; skills can also package workflows, knowledge, constraints, procedures, or examples.Source: §6 · p. 8
  • Prior reusable-experience systems acquire or refine skills, whereas this paper assumes an existing library and tests whether agents can route to skills as it grows.The cited examples include executable code skills, verbal feedback, natural-language lessons, web workflows, and hierarchical skill banks.Source: §6 · p. 8
  • Long-context studies examine failures to use relevant information, but this paper reports that masking irrelevant skill metadata largely restores loading, implicating semantic and instruction interference rather than length alone.The passage contrasts this result with prior work reporting effects of context length even with perfect retrieval and masked irrelevant tokens.Source: §6 · p. 8

7 Conclusion

The paper concludes that growing skill libraries lead agents to load fewer or no relevant skills, and that neither alternative-skill loading nor planning prompts explain away the problem. It argues for explicit routing and evaluation of skill use, while noting preselection’s coverage and latency trade-offs.

7 Conclusion

  • As libraries grow, agents load fewer or no skills despite relevant skills remaining available, and loading other skills is too rare to explain the decline.The conclusion identifies semantic interference from descriptions and instruction interference when skill choice competes with task completion.Source: §7 · p. 8
  • Masking irrelevant descriptions largely restores skill loading, while planning skill use with metadata intact remains less effective than preselection.Preselection can stabilize skill use but introduces coverage and latency trade-offs.Source: §7 · p. 8
  • The conclusion recommends explicit routing and evaluating skill use alongside task success; a future direction is revisiting selected skills during execution to recover omissions from fixed preselection.Source: §7 · p. 8

Limitations

The study’s scope is bounded by a budget-limited proprietary-model check, task exclusions, and controlled interventions on a 10-task subset. The appendices also document the benchmark, licensing, and execution environment.

Limitations

  • Proprietary-model validation covers only 37 tasks and not every routing setting, so it serves as a consistency check rather than a second full benchmark.The study also cautions that masking results on a smaller subset and fewer repetitions do not establish equivalence between conditions.Source: Limitations · p. 9

A SkillsBench Dataset Details

A.1 License compliance

  • The authors use the repository under Apache License 2.0, without modifying or redistributing the benchmark, and preserve attribution through citation.They describe this use as consistent with the license’s permissions and notice requirements.Source: §A.1 · p. 12

A.2 Server configuration

  • Main experiments ran on Linux servers with an AMD EPYC 7K62 and 62.3 GiB of memory; field-level masking ran separately on an Apple Silicon machine.The masking environment’s 16 GB of memory is specified, and its measurements are not used for latency comparisons.Source: §A.2 · p. 12

B Main Study Task Exclusions

C Experimental Subset Tasks

  • Budget-limited controlled interventions use the same 10-task subset for original, masked, plan-first, and explicit-preselection comparisons.A valid implicit-routing run averages more than 15 minutes and roughly half a million input tokens even without irrelevant skills, motivating the smaller subset.Source: C Experimental Subset Tasks · p. 12, A SkillsBench Dataset Details · p. 12

D Full Data for Scaling Curves

The appendix supplies tabular values for the scaling curves and clarifies how implicit-routing and routing-strategy results are organized. Its comparison shows that skill-routing metrics deteriorate earlier and more sharply than task success.

D Full Data for Scaling Curves

  • The appendix provides exact values behind the scaling curves, with the full model-by-strategy Pass@3 matrix already reported in Table 3.The additional tables cover plotted slices and skill-loading measurements not otherwise shown in full.Source: D Full Data for Scaling Curves · p. 13, Table 13 · p. 13

D.1 Implicit-routing scaling by execution model

  • Routing metrics deteriorate earlier and more sharply than Pass@3 as irrelevant skills grow, including when GLM-5 task success stays near its small-library value at 100 irrelevant skills.This motivates treating loaded-skill recall and zero-load rate as first-class metrics rather than relying only on verifier success.Source: §D.1 · p. 13, Table 15 · p. 13, Table 16 · p. 13

D.2 Skill scaling by routing strategy

  • Tables 17–19 report aggregate Pass@3, loaded-skill recall, and zero-load rate across routing strategies and library sizes, complementing Table 3’s task-success breakdown by execution model.The strategy-level tables add the two skill-use metrics.Source: §D.2 · p. 13, Table 17 · p. 13, Table 18 · p. 13

E Budget-Limited Proprietary Model Validation

A budget-limited validation finds the same library-growth pressure in proprietary models, while appendices distinguish runtime and preselection precision and examine task demand and hybrid candidate aggregation. The results show that stable task success can coexist with substantial routing degradation.

E Budget-Limited Proprietary Model Validation

F Precision Metrics

  • Loaded-skill precision is computed only over runs that load at least one skill, whereas selected-skill precision is computed over nonempty preselected sets.Because agents increasingly load fewer or no skills as libraries grow, loaded precision usually declines less sharply than recall.Source: F Precision Metrics · p. 14

G Skill Demand Breakdown

  • At 1,000 irrelevant skills, Naive exposure loads fewer golden skills than explicit-routing strategies across all three task-demand bins, and partial retrieval can miss a workflow needed for passing multi-skill tasks.The demand bins are single-skill, two- or three-skill, and four-or-more-skill tasks.Source: G Skill Demand Breakdown · pp. 14–15

H Hybrid Candidate Aggregation Details

  • Quota aggregation gives the strongest all-gold coverage at 10,000 irrelevant skills, using a fixed number of top unseen skills from each query to encourage coverage of different sub-procedures.The paper uses Quota as its default aggregator.Source: H Hybrid Candidate Aggregation Details · p. 15

I Non-Golden Skill Loading

This section examines non-golden skill loading and skill-selection approaches, including preselection’s coverage and routing-delay trade-offs and a plan-first intervention that retains the original skill metadata. It also describes execution and prompting procedures and discloses the use of AI assistants in writing and parts of the experimental code and analysis.

I Non-Golden Skill Loading

  • Non-golden loads occur in 2.32% of evaluations across scales, and potential alternatives above the threshold occur in 0.95%; both remain far less common than the decline in skill use.Source: I Non-Golden Skill Loading · p. 15

J Preselection Trade-offs and Latency

  • As distractors grow, each preselection method retains fewer golden skills, and reduced coverage limits task success, though aggregate coverage alone does not determine outcomes.RAG can retain more golden skills than LLM preselection yet yield lower task success, depending on which skills survive selection.Source: J Preselection Trade-offs and Latency · p. 16
  • From 10 to 1,000 distractors, median routing delay rises from 8 to 19 seconds for LLM preselection and from 33 to 41 seconds for Hybrid.The methods ran in different periods, so this is not a controlled latency comparison; routing delay also need not predict end-to-end wall time.Source: J Preselection Trade-offs and Latency · pp. 16–17, Table 31 · p. 17

K Execution and Routing Algorithms

  • Execution exposes preselected skill metadata, then iterates model tool calls before verification; Hybrid generates queries, retrieves candidates, and makes a final selection.The baseline preselectors return all skills, use an LLM to select from full metadata, or rank by embedding similarity and return top-k skills.Source: K Execution and Routing Algorithms · pp. 17–18

L Prompt Templates

  • Execution prompts require checking available skills and explicitly loading a skill before use, while preselection prompts return skill names as JSON.Hybrid query generation asks for concise queries covering distinct task capabilities, including intermediate capabilities that the task may not emphasize.Source: L Prompt Templates · p. 18, §L.1 · p. 18, §L.2 · p. 19

L.5 Intervention prompts

  • Field-level masking changes only designated distractor fields, leaves golden metadata readable, and serves as a targeted diagnostic rather than a test of statistical equivalence.The separate masked intervention replaces irrelevant skills’ names, descriptions, and locations with approximately original-length placeholders while keeping golden metadata readable.Source: §L.5 · p. 19
  • The plan-first prompt retained the original skill metadata and added an instruction to assess skill relevance before executing commands.It asked agents to state their reasoning, load relevant skills, and only then begin solving the task.Source: §L.5 · p. 20

M Use of AI Assistants

  • AI assistants supported writing and parts of the experimental code and analysis, with authors reviewing and editing the resulting text and code.Source: M Use of AI Assistants · p. 20

N Per-task Scaling Heatmaps

The per-task heatmaps show many tasks becoming less successful as irrelevant skills accumulate, while explicit preselection stabilizes many of those same tasks; retrieval-based approaches show mixed outcomes.

N Per-task Scaling Heatmaps

Related papers

Loading 2610.18522v1…