Source-linked AI summary
Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness
Kevin Dela Rosa
TL;DR
Agent harnesses must select the skill matching a user task, but the effect of exposing skill bodies versus listings is unclear at small scale. This paper studies Tinycloud’s representations and exposure regimes, finding that full exposure succeeds while partial exposure can misroute through lexical competition and all-off can cause costly discovery failures.
Problem
The paper asks how prompt-side skill representations affect in-context selection in a production harness with a growing skill library.
Method
It presents a Tinycloud case study of tool-skills, workflow-skills, and their full-body versus listing-only exposure across three ablated regimes.
Results
Full autoload selects the gold skill on every task, while all-off produces hard discovery failures and the default misroutes one task through lexical competition.
Takeaways & Limitations
Skill authors should treat semantically overlapping workflow and autoloaded tool names as a routing risk, not merely a context-budget trade-off.
Takeaways & Limitations
The evidence is limited to one harness with thirteen skills and six fixture tasks; real logs and larger libraries remain future work.
Abstract
from arXiv · showhide
A production agent harness must discover and rank, from a growing library of skills, the one most appropriate for a user's task. At small scale this selection happens in context: the LLM planner chooses among skill representations exposed in its system prompt, without an explicit embedding-based retrieval step. We treat this in-context selection as the small-N counterpart to embedding-based skill retrieval at scale, and present a case study of how Tinycloud, a production multimodal video agent harness, represents its skills for the planner. The harness ships skills under two recurring representations: tool-skills that wrap a single external API or system tool and serve as primitive vocabulary, and workflow-skills that orchestrate tool-skill calls plus a template render to produce one named deliverable. The harness exposes them via two surfaces in the system prompt: an inlined-body surface (full instructions, scripts, templates) for autoloaded skills, and a one-line listing for on-demand skills. A six-task selection ablation across three exposure regimes (all-on, default, all-off) shows that full autoload selects the gold skill on every task; all-off slows execution and produces hard discovery failures; and the production default misroutes one task because its lexical signal collides with an autoloaded tool-skill that pulls planner attention away from a listed workflow-skill. The headline finding is that in-prompt exposure of skills is not monotonically helpful: partial exposure can create lexical competition that suppresses correct selection. We connect this small-N observation to recent retrieval-based skill-routing work at large scale, and frame this contribution as a case study rather than a benchmark.
1 Introduction
This case study treats skill selection among 13 prompt-exposed skills as in-context retrieval and examines how representation surfaces affect that selection. Tinycloud distinguishes tool-skills from workflow-skills and evaluates their exposure through a six-task, three-regime ablation.
- Problem framing: At 13 prompt-exposed skills, the LLM planner performs in-context selection rather than explicit embedding-based retrieval.The paper treats this as the small-N counterpart to learned skill ranking at larger scale.
- Representation: Tinycloud exposes skills either as full inlined bodies for autoloaded skills or one-line listings for on-demand skills.The paper reframes this conventionally context-budgeted choice as a representation choice for selection.
- Skill taxonomy: Tool-skills provide autoloaded primitive vocabulary, whereas workflow-skills orchestrate tool calls and template rendering into named deliverables.This distinction maps onto the two prompt exposure surfaces.
- Evaluation: The study runs a six-task selection ablation across all-on, default, and all-off exposure regimes, measuring correctness, wall-clock cost, and tool-call count.It presents the work as a production case study rather than a benchmark.
- Headline finding: In-prompt exposure is not monotonically helpful because partial exposure can create lexical competition that suppresses correct selection.The paper identifies this as a central observation and contribution.
2 Background
The paper situates its small-N in-context selection study within research on retrieval over skills, tools, and composable agent capabilities. It contrasts Tinycloud’s author-curated deliverable workflows with broader systems designed for larger-scale discovery and memory management.
- Agent and tool retrieval: SkillRouter learns retrieval-based routing over ∼80k skills, while ToolLLM catalogs over 16k real-world APIs for large-scale tool selection.These works represent the closest concurrent lines of skill and tool retrieval research identified by the paper.
- Agent and tool retrieval: Prior work also covers foundational tool use, interleaved reasoning and acting, skill efficacy benchmarking, and multimodal-retrieval routing.The cited lines include Toolformer, ReAct, SkillsBench, and multimodal retrieval studies.
- Skill format, composition, and memory: Tinycloud’s workflow-skills resemble Voyager’s composable code chunks but are author-curated, deliverableshaped, and template-emitting.The comparison distinguishes the paper’s workflow representation from automatic curriculum-based discovery.
- Skill format, composition, and memory: The autoload flag is a static, design-time analog of MemGPT’s runtime context paging.Both frame context as managed memory, but the paper studies the static prompt-side choice.
3 The Skill System
Tinycloud represents each skill as a directory and loads it through either a detailed autoload surface or a lightweight on-demand listing. Workflow-skills act as recipes that compose tool-skills and render a user-facing deliverable.
- 3.1 Format and loader: Each Tinycloud skill directory contains SKILL.md metadata and procedural knowledge, optional executable scripts, and optional reference templates.SKILL.md includes YAML frontmatter such as name, description, and optional autoload.
- 3.2 Surface area in the system prompt: Autoloaded skills appear with full details and paths in the system prompt, while on-demand skills expose only name and description until invoked.The asymmetry limits context use for workflows that fire only occasionally.
- 3.2 Surface area in the system prompt: Tool-skills are atomic primitives, whereas workflow-skills combine multiple tool calls, reasoning, and template rendering for explicitly requested deliverables.The agent invokes the workflow once, while its orchestration script handles underlying tool-skill calls.
- 3.4 Templates as a structural fingerprint: Workflow-skills consistently contain one deliverable template, while tool-skills generally contain none or a reusable primitive template library.Template count therefore helps distinguish deliverable recipes from primitive tools.
4 Catalogue
The catalogue shows a consistent structural split between Tinycloud’s tool-skills and workflow-skills. Category, autoload status, script count, and template count together expose whether a skill is primitive vocabulary or a deliverable recipe.
- Catalogue structure: Every workflow-skill is on-demand and contains exactly one template, while every tool-skill is autoloaded and contains zero templates or one primitive template library.This is the catalogue-wide split reported for the twelve production skills.
- Structural fingerprint: All five workflow-skills share two scripts and one deliverable template because they use the same orchestration-and-rendering recipe.The scripts are an orchestrator and a renderer.
- Structural fingerprint: Tool-skills vary in script complexity but uniformly lack deliverable templates, serving instead as composable vocabulary.The exception is media-artifact, whose seven templates form a reusable primitive library consumed by workflows.
5 Empirical
The empirical study measures prompt-token budgets and tool selection across All-on, Default, and All-off loading conditions. Full exposure selects the gold skill on every task, while partial or absent exposure introduces routing failures and higher execution costs despite modest token-budget differences.
- Experimental setup: The ablation compares All-on, production Default, and All-off over the marginal effect of 13 skills, with one headless Tinycloud run per condition–task pair.The study uses six fixture tasks, no paraphrase or seed sweep, and records invoked skill and canonical-deliverable production.
- Token-budget findings: The entire workflow-skill bundle adds approximately 378 tokens versus Default, while static-rule overhead of approximately 6.4k tokens dominates the total prompt.The skills block itself varies by approximately 4×, but All-on remains within the same order of magnitude as All-off.
- Routing findings: All-on selects the gold skill on all six tasks, whereas Default misses q3 and All-off misses q2 and q5.The table defines ✓/✗ as gold-skill invocation; canonical deliverables were produced in every ✓ cell and no ✗ cell.
- Measurement: Token ratios use tiktoken cl100k_base as a public-tokenizer proxy for Claude Opus’s tokenizer, and are insensitive to tokenizer choice.The token counts come from Tinycloud’s system-prompt builder under each loading condition.
- Routing findings: Default and All-on have roughly tied mean wall-clock time, while All-off is approximately 1.8× slower because the agent searches the filesystem for skill scripts.In q5 under All-off, this escalated to subagent delegation, costing 17 minutes and 83 tool calls.
6 Discussion
The discussion argues that partial prompt exposure can harm skill routing through lexical competition, while autoloading reduces discovery variance and supports practical skill-authoring heuristics. It situates these findings as a small-N case study whose broader generality and scaling implications remain open.
- Lexical competition: Default misroutes q3 because the inlined video-analyze body attracts planner attention through Analyze, while listed ad-analysis lacks its full body.All-on and All-off route q3 correctly, making Default the only condition with this asymmetric exposure.
- Exposure regimes: All-off produces two outright failures and imposes a 35–170% wall-clock premium on other successful cells.Q2 yields no deliverable, while Q5 bypasses the gold tool after 17 minutes and 83 tool calls.
- Skill authoring: A practical authoring heuristic is to use workflow-skills for named single-artifact deliverables and tool-skills for primitives without a single deliverable shape.In this harness, workflow-skills are on-demand and tool-skills are autoloaded vocabulary.
- Composition: Workflow-skills are leaves that call tool-skills, while higher-level workflows prefer tool-skill chains over inner workflow-skills whose deliverable shape may not fit the outer task.The paper gives a quarterly review report from N sales calls as an example.
- Generality: The evidence is limited to one harness, so the proposed distinction beyond multimodal video agents remains a hypothesis.The paper names audio production, GIS, and computational biology as possible future domains.
- Scope and scaling: Embedding-retrieval baselines are omitted because the study examines prompt-side representations at thirteen skills, while learned retrieval becomes relevant as libraries outgrow the prompt.The authors identify q3-style lexical competition as a possible failure mode for larger retrieval systems as well.
- Future work: Future work should test lexical competition at larger library sizes, in production task logs, and with learned per-prompt autoload policies.These extensions are explicitly out of scope for the present case study.