PaperlayerView exact revision on arXiv

Source-linked AI summary

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

Ruishuo Chen, Xun Wang, Yu Chen, Zhuoran Li, Longbo Huang

arXiv:2609.15982v1cs.LGcs.AIcs.CL

TL;DR

Skill routing must preserve both context efficiency and the agent LLM’s task understanding. Gavel reads routing signals from a frozen model using two trained projections, then combines a library-wide glance with a full-attention verdict. It transfers across benchmarks and outperforms contextual and external-router baselines, including when skills become necessary mid-rollout.

  • Problem

    Existing routers either crowd the agent’s context with all skill metadata or move selection to external models that lack the agent’s own capability.

  • Method

    Gavel uses two trained linear maps to read task and skill mid-layer states, shortlist skills with a glance, and fuse the frozen model’s likelihood and yes/no judgment in a verdict.

  • Results

    Across three public benchmarks and 372-trajectory SkillTraj, Gavel outperforms progressive disclosure and retrieve-and-rerank systems by up to 13.4 points on written tasks and 21.9 points mid-rollout.

  • Takeaways & Limitations

    A lightweight read-out can put routing capability already present in a frozen agent LLM to work without adding a standalone model or skill text before selection.

  • Takeaways & Limitations

    The design assumes task-only context, a frozen agent model, and installation-time indexing without training for each new skill.

Abstract

from arXiv · show

Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. Deployed harnesses route by preloading every skill's metadata into the context, which disperses the agent's attention and caps the library size. Retrieval pipelines move the selection out of the context, but also out of the agent's capability. We show that the frozen agent LLM already carries the routing signal in its own forward passes, and that two linear maps suffice to read it out with no skill text in the context. Gavel (Glance And Verdict from a frozen LLM) reads it in two steps. A glance projects the task's and each skill's mid-layer states through the two maps, the only parameters trained, and scores the full library against compact per-skill banks that one forward pass builds at installation. A verdict then resumes the shortlisted skills' forward passes and reads the model's own likelihood and yes/no judgment, fused with the glance as a product of experts. Trained once, Gavel transfers zero-shot to three public benchmarks and SkillTraj, our new benchmark of 372 simulated agent trajectories. On Qwen3-32B it outperforms progressive disclosure and retrieve-and-rerank pipelines that add 1.2B to 16B external parameters, by up to 13.4 points on written tasks and up to 21.9 when the need for a skill arises mid-rollout. Routing accuracy improves as the backbone does, and in a bash-agent harness the same 32B triggers the correct skill on Skill-Use more often than far larger frontier models running in Codex.

1. Introduction

Skill routing must balance context efficiency with the agent LLM’s task understanding. Gavel addresses this tension by reading routing signals from the frozen model’s own forward passes and outperforms contextual and external-router alternatives across written and mid-rollout evaluations.

  • Motivation and approach: Gavel resolves the central routing trade-off by using the frozen agent LLM’s forward passes without loading skill text until selection.This avoids both context crowding from progressive disclosure and capability separation from external retrieval systems.
  • Results: Gavel is trained once and transfers zero-shot across benchmarks, while routing accuracy improves with the backbone and Qwen3-32B outperforms larger frontier models in a bash-agent harness.The projections are the only trained parameters in the described instantiation.
  • Results: 13.4 points over the strongest pipeline on SRA-Bench, with 8.6 to 21.9 points across SkillTraj’s four scenarios.The comparisons are against progressive disclosure with a twenty-skill shortlist and retrieve-and-rerank systems adding 1.2B to 16B external parameters.
  • Method: Gavel uses a token-level glance through two trained linear maps, followed by a verdict that combines generative and discriminative evidence as a product of experts.The glance ranks the full library, while the verdict examines only shortlisted skills with the frozen model’s likelihood and yes/no judgment.
  • Evaluation: SkillTraj evaluates routing when a skill becomes needed during 372 simulated agent trajectories under noisy multi-turn context.This complements three public benchmarks whose inputs are written tasks.

2. Related Work

Prior skill and tool-selection systems either expose all metadata to the agent or delegate selection to external models. Related work therefore frames Gavel against context-based routing, retrieval stacks, and methods for reading information from frozen or adapted language models.

  • Skill and tool selection: Progressive-disclosure harnesses keep every skill’s metadata in context for the agent, while retrieval systems select skills externally from APIs or full documents.These approaches represent the two established routes that Gavel seeks to bridge.
  • Skill and tool selection: ToolkenGPT emits tools as tokens but requires one embedding per tool, whereas Gavel reads routing signals from the frozen model and avoids a training run for each new skill.The latter distinction concerns installation-time extensibility.
  • Reading information out of an LLM: Prior read-out methods probe hidden states, fine-tune decoder LLMs into encoders, or unify embedding and generation by adapting the backbone.Gavel instead trains only a lightweight read-out over the frozen agent model’s states.

3. Gavel: Routing on the Agent’s Own Forward Passes

Gavel routes skills from the agent’s own hidden states without preloading skill text: a compressed installation-time key bank supports a library-wide glance, and full-attention verdicts rescore shortlisted skills. Its three signals are combined as estimates of the same skill posterior.

  • 3.1. Problem setup and design principles: Gavel satisfies task-only context, no-standalone-model, and installation-time-indexing requirements for changing skill libraries.Skill text enters the context only after a skill is chosen, selection inherits the frozen agent’s capability, and new skills require no training run.
  • 3.2. The glance: token-level read-out of the frozen forward pass: The glance maps task and skill-token states from one intermediate layer into queries and keys, then each task token keeps its strongest match against every skill.Skill keys are built with one installation-time forward pass, and the full-library sweep uses token-level voting rather than span averaging.
  • 3.3. Compressed skill banks: The skill banks use ε-covers to remove redundant keys while lowering each score by at most ε and inflating none.Farthest-first construction makes bank size depend on ε-distinguishable directions rather than document length.
  • 3.4. The verdict: native signals under the model’s full attention: The verdict resumes each shortlisted skill’s forward pass with the task, measuring mean task log-likelihood and the model’s yes-over-no log-odds.Because the yes/no question follows the task, one causal pass yields both read-outs without altering task-position predictions.
  • 3.5. The ruling: a product of experts: With a uniform library prior, the glance, likelihood, and judgment can each be interpreted as estimates of the same log posterior over skills.This shared interpretation motivates the product-of-experts ruling, where any one expert can veto candidates tolerated by the others.

4. Experiments

Gavel is evaluated against retrieval, reranking, progressive-disclosure, ablation, mid-rollout, deployment, and backbone-scaling baselines. It leads the reported written-task and SkillTraj comparisons, triggers the correct skill in more than 90% of Skill-Use tasks, and improves with backbone capability.

  • 4.2.1 Comparison with current practice: 3.8, 13.4, and 1.3 to 2.7 points: Gavel’s full pipeline beats the strongest pipeline on SkillRet, SRA-Bench, and Eval-Core, respectively.Gavel also leads all three benchmarks at the retrieval and full-pipeline stages.
  • 4.2.1 Comparison with current practice: 13.4 points: dense retrievers trained on similar data lag after query style or skill-document genre shifts, while Gavel’s glance remains strongest on SRA-Bench and Eval-Core.On SkillRet, the glance comes within a point of the best embedder, while trained embedders match it on that in-domain benchmark.
  • 4.2.2 Ablations: Every benchmark: replacing Gavel’s glance with Jina-ColBERT-v2 loses badly, showing the advantage comes from the frozen agent LLM’s compressed representations.The replacement preserves fine-grained token-pair scoring but removes the agent LLM read-out.
  • 4.2.2 Ablations: Every benchmark: using document likelihood in the verdict falls visibly behind, because body likelihood reflects writing resemblance more than task usefulness.Calibration discards document likelihood, setting α to zero, while the remaining signals still underperform.
  • 4.3. Routing mid-rollout: 8.6 to 21.9 points: Gavel leads every SkillTraj scenario over the strongest alternative during noisy mid-rollout routing.Progressive disclosure holds up on wrong-skill recovery but collapses on agent plans whose relevant details require the skill body.
  • 4.5. Scaling with backbone capability: 12.2 points: on Qwen3.8-27B, progressive disclosure still trails Gavel on the same backbone, while Gavel’s routing accuracy rises with model size and generation.Gavel on a 0.6B model already beats 32B progressive disclosure in the cited SRA-Bench comparison.

5. Conclusion

Gavel returns skill selection to the frozen agent LLM without placing skill text in context, using a learned read-out over native forward-pass signals. It outperforms progressive disclosure and external retrieval pipelines across the reported benchmarks, while leaving broader routing applications open.

  • Only two linear projections are trained, and the routing signal is extracted without placing skill text in the model context.
  • Across three public benchmarks and SkillTraj, Gavel outperforms progressive disclosure and retrieve-and-rerank pipelines that add up to 16B external parameters.
  • In a live harness, Qwen3-32B loads the correct skill more often than far larger frontier models running in Codex.
  • Gavel’s glance scores the full library from frozen mid-layer states, while its verdict re-scores shortlisted skills using generative and discriminative predictions.
  • The appendix documents read-out failures, layer selection, projection training, compression guarantees, cross-backbone evaluation, ablations, prompts, rollout transcripts, routing cost, and gating.

A. Native read-outs that fail

Native attention machinery alone does not provide a reliable routing space, motivating the learned glance’s two projections and hand-crafted aggregation. The learned retrieval-token alternative also fails to allocate attention selectively on the available training data.

  • The appendix’s alternatives each fail in a way that informs the adopted design rather than replacing its learned read-out.
  • Native attention keys reach Hit@1 .001 on SkillRet versus .918 for the trained glance head, with exact zeros on SRA-Bench.
  • Native initialization finishes 0.7 points below random initialization, while the multi-head shape is 0.6 points below the single wide head.
  • Trained retrieval tokens learn recency rather than content-selective aggregation because clean single-turn requests make recent context sufficient during training.
  • The adopted glance therefore retains a single randomly initialized head and hand-crafted decay, leaving learned aggregation for data that rewards selectivity.

B. Layer selection

The read-out layer is selected near the backbone’s compression valley, where hidden-state entropy reaches a floor. A trained scan confirms the layer chosen by the unsupervised entropy criterion, while raw cosine similarity does not.

  • The entropy of hidden states rises through the first two thirds of depth before collapsing from 4.01 to 1.83 across a single block.
  • The entropy measure removes the top-norm sink token so it reflects representation structure rather than the massive activation dominating the spectrum.
  • Layer 44 comes first on R@20 among six trained candidate depths, matching the entropy-floor criterion shown in Figure 7.
  • Raw cosine reverses the trained result: layer 44 reaches R@20 .503, versus .916 at layer 60, across a 44-point depth spread.

C. Training details

Gavel trains only two projection matrices while freezing and caching the backbone’s hidden states. Training uses a fixed contrastive recipe, followed by a separate compressed-key stage and rollout-specific vote decay.

  • The two bias-free projections are 768 × 5120 matrices totaling 7.9M parameters, trained while gradients stop at cached backbone states.
  • Projection training uses SkillRet, temperature 𝜏= 40, fp32 AdamW, learning rate 10−3, weight decay 0.01, and 24,000 steps.
  • Compressed keys are trained for 3,000 steps at learning rate 10−4 with the projections and banks frozen.
  • Written-task benchmarks use uniform vote weights, while the decay applies only during mid-rollout routing.

D. Compression guarantees

Compression replaces each skill’s token-level key bank with a compact, geometrically controlled subset while preserving routing scores up to explicit error bounds.

  • The score bound follows by covering the best key and applying Cauchy–Schwarz to unit-norm query and key vectors.The proposition compares the maximum query–key score over the full bank with the maximum over its ε-cover.
  • Only tokens near a skill’s top-k boundary can change vote membership under compression; their total weighted contribution supplies the additional stability term.The boundary set B_x(s) contains tokens whose skill score lies within ε of the competing kth-largest score.
  • Compression lowers every key score by at most ε and never raises it, with vote stability inherited for skills away from top-k boundaries.When a skill clears every top-k boundary by more than ε, its membership is unchanged and its vote decreases by at most ε.
  • The worst-case bound is loose for routing-relevant keys because tightness requires a query that barely matches its best key.On tokens deciding routing, the query is close to its best key, making the extremal Cauchy–Schwarz configuration unlikely.
  • An ε-cover that is also ε-separated yields bank sizes between the covering and packing numbers of the skill’s key set.The retained bank is both an ε-cover and an ε-separated set, so N_ε(D_s) ≤ |C_s| ≤ P_ε(D_s).

E. A non-Qwen backbone

Rebuilding Gavel on Gemma-4-31B preserves its benchmark-leading pattern, while exposing that full skill bodies help one benchmark but can hurt others.

  • A non-Qwen backbone: Gavel again achieves the highest adjudicated score on all three benchmarks after rebuilding the pipeline for Gemma-4-31B.The backbone, chat template, read-out depth, and projections are changed or retrained rather than transferred from the Qwen build.
  • Full-body progressive disclosure: Seven-point gains from showing twenty full skill bodies occur only on SRA-Bench, while SkillRet falls 2.2 points and Eval-Core changes by one to three queries.The bodies fit within Qwen3-32B’s 131K-token window, with the longest prompt reaching 124K tokens.
  • Full-body progressive disclosure: Gavel remains 16 adjudicated points ahead on SRA-Bench and 10 points ahead on SkillRet despite the expanded progressive-disclosure prompts.
  • Verdict ablation: Combining the glance with both verdict signals is best on every benchmark, exceeding the stronger single-signal variant by 1.9, 3.8, 4.6, and three queries across the evaluated pools.The comparison pools SkillRet, SRA-Bench, SkillTraj, and the two Eval-Core pools as reported.

F.3. Replacing the glance with an off-the-shelf embedder

Replacing Gavel’s glance with an off-the-shelf embedder preserves the verdict stage but weakens the overall router, showing that the native read-out contributes beyond shortlist reranking.

  • Experimental setup: The replacement experiment keeps the embedder, reranker, glance, and full ruling as separate reference systems while calibrating the ruling on validation data.
  • Replacing the glance with an off-the-shelf embedder: Gavel stays ahead of the embedder-based variant on every benchmark: 2.9 points on SkillRet, 2.2 on SRA-Bench, one query per Eval-Core pool, and 9.1 on SkillTraj.The embedder receives the last message on SkillTraj, while the verdict passes remain unchanged.
  • Verdict on an external shortlist: The verdict improves the embedder’s shortlist by 9.0 points on SkillRet, 24.4 on SRA-Bench, and 19.9 on SkillTraj.On the same shortlist, it also beats Qwen3-Reranker-8B by 5.2, 11.2, and 8.6 points on the three larger libraries.
  • Why the glance matters: The native glance ranks skills from states the agent already produces, whereas an embedder requires a separate task pass before ranking.The glance therefore adds a bank sweep rather than another model forward pass over every task.

H.1. Under-annotated golds: examples from all three benchmarks

The appendix shows that raw Hit@1 can undercount adequate skills because libraries contain duplicates and useful distractors, motivating adjudication and validating its broad score pattern.

  • Under-annotated golds: Raw Hit@1 counts only annotated golds, so a router can be marked wrong when it selects a different skill that fully serves the task.The three libraries contain repeated or overlapping capabilities, making exact identifier matching stricter than adequacy.
  • SkillRet: In SkillRet, the router’s alternative mcp-builder copy is preferred over the annotated gold because the gold is described as generic and outdated.Both documents serve the MCP-server task, but only one is listed as the annotation.
  • SRA-Bench: In SRA-Bench, a distractor-pool scikit-learn skill is credited by adjudication because it adequately serves the SVM-classification task.The curated golds include scikit-learn and warnings skills, but the returned distractor is judged adequate in all four comparisons.
  • Eval-Core: In Eval-Core, a near-verbatim rdkit duplicate receives a raw miss even though the judge finds both documents essentially identical.The duplicate shares its description word for word and has body 4-gram Jaccard .92 with the gold.
  • Adjudication reliability: Adjudication lifts every system, yet the full Gavel pipeline remains above every other row on raw Hit@1 across all three benchmark tables.Human checks agree with the judge on 171 of 200 pairs, or 86%.
  • Adjudication caveat: The judge’s crediting rule is necessary but imperfect because skill documents can contain thousands of tokens and decisive details deep in their bodies.A disagreement with human annotation therefore does not by itself establish which evaluator erred.

I. Deployment walkthroughs

Deployment walkthroughs show Gavel triggering skills during live agent rollouts, including when the need emerges mid-message, while progressive disclosure rarely triggers skill use without prompting.

  • Live rollout examples: Gavel can fire after an initial failed attempt, interrupting the rollout and loading pdftk-server before completing the editable W-4 form.The restarted turn re-issues the command on the found path and preserves the form’s editability.
  • Live rollout examples: It also triggers just before an endpoint migration is named, loading vercel-ai-sdk and guiding replacement of v5 generateObject with generateText plus an Output.object schema.The eventual edit follows the loaded skill’s Vercel AI SDK v6 patterns while preserving the caller-facing contract.
  • Live rollout examples: A deeper trigger occurs midway through a third-turn message after the agent reads instructions and dumps a form, showing that firing need not occur at message boundaries.The appendix notes that the gate is read at every decoded token, so a firing interrupts the message wherever it occurs.
  • Live rollout examples: In a connector-hardening trajectory, Gavel fires twice: first for the gold connector skill and later for a genuinely needed non-gold skill after OAuth configuration enters context.This illustrates multiple skill loads within one task as the rollout state changes.
  • Progressive disclosure versus Gavel: The same Qwen3-32B backbone reaches .909 skill-trigger rate with Gavel, versus .834 when forced to read the selected skill and only 2 of 175 tasks under progressive disclosure.The benchmark’s failure is deciding to consult a skill at all, rather than choosing among listed skills.

J. The cost of routing

Gavel shifts routing computation off the rollout context, trading a small batched verdict cost for reduced persistent attention over skill metadata. Its cost is nearly independent of library size and session length, while latency is concentrated in each verdict firing.

  • Cost comparison: Gavel’s estimated routing bill is approximately $0.080 m and is independent of n and T under the nine-forward pruning margin.The glance and gate reuse rollout states, while the verdict uses nine batched prefills per firing.
  • Cost comparison: Three Gavel firings cost $0.24, while a hundred-skill menu over fifty calls costs $0.25; Gavel becomes cheaper once n exceeds roughly 2,000 m/T.Which router is cheaper depends on the number of firings per session, session calls, and library size.
  • Latency: Gavel’s latency is concentrated in roughly half a second per firing for nine parallel prefills of about 2.2K tokens, decreasing with per-skill KV caching.The gate and glance add only state reading and a matrix multiply, whereas progressive disclosure adds no separate routing stage.
  • Context and serving costs: Progressive disclosure retains 100n metadata tokens in every session prompt, repeatedly resending them across T calls and keeping their attention scan on every decoded step.Prompt caching avoids recomputing the menu but does not remove its persistent context occupancy or attention cost.
  • Deployment considerations: The current gate fires after remaining on for two consecutive tokens, and the router can be hosted by the model provider with projections shipped per backbone.The authors describe this timing rule as unoptimized and intended to test whether routing signals survive live rollouts.
Loading 2609.15982v1…