Source-linked AI summary
SkVM: Revisiting Language VM for Skills across Heterogenous LLMs and Harnesses
Le Chen, Erhu Feng, Yubin Xia, Haibo Chen
TL;DR
Existing skill systems pass natural-language skills directly to models, producing inconsistent behavior across heterogeneous LLMs and harnesses and reducing portability and efficiency. SkVM treats skills as code, decomposes requirements into primitive capabilities, and compiles and runs them with environment binding, concurrency extraction, and runtime optimization. Across eight LLMs and three harnesses, it improves task completion while reducing token use by up to 40%, with reported speedups and latency reductions from parallelism and code solidification.
Problem
Current systems treat skills as raw context, leading to inconsistent execution across models and harnesses and limiting skill portability and efficiency.
Method
SkVM uses capability-based compilation, environment binding, concurrency extraction, JIT code solidification, and adaptive recompilation to execute skills across heterogeneous targets.
Results
SkVM improves task completion across eight LLMs and three harnesses, reduces token consumption by up to 40%, and achieves 3.2×–50× wall-clock speedups.
Takeaways & Limitations
Skill compilation provides a systematic pathway for transforming fragile prompts into portable components across diversifying models and agent harnesses.
Takeaways & Limitations
The current catalog contains 26 primitive capabilities covering the 15,063 analyzed skills, and must expand as the skill library grows.
Abstract
from arXiv · showhide
LLM agents increasingly adopt skills as a reusable unit of composition. While skills are shared across diverse agent platforms, current systems treat them as raw context, causing the same skill to behave inconsistently for different agents. This fragility undermines skill portability and execution efficiency. To address this challenge, we analyze 118,000 skills and draw inspiration from traditional compiler design. We treat skills as code and LLMs as heterogeneous processors. To make portability actionable, we decompose a skill's requirements into a set of primitive capabilities, and measure how well each model-harness pair supports them. Based on these capability profiles, we propose SkVM, a compilation and runtime system designed for portable and efficient skill execution. At compile time, SkVM performs capability-based compilation, environment binding, and concurrency extraction. At runtime, SkVM applies JIT code solidification and adaptive recompilation for performance optimization. We evaluate SkVM across eight LLMs of varying scales and three agent harnesses, covering SkillsBench and representative skill tasks. Results demonstrate that SkVM significantly improves task completion rates across different models and environments while reducing token consumption by up to 40%. In terms of performance, SkVM achieves up to 3.2x speedup with enhanced parallelism, and 19-50x latency reduction through code solidification.
1 Introduction
Skills are increasingly used as reusable natural-language programs, but treating them as raw context makes execution inconsistent across heterogeneous models and harnesses. SkVM addresses this mismatch with compilation and runtime support that tailors skills, manages execution, and improves efficiency.
- Motivation: Over 100,000 skills now support agent tasks, but current systems pass them directly to models as additional context.Skills encapsulate domain-specific procedures and best practices through natural-language descriptions and scripts.
- Motivation: 15% of tasks degrade when skills are enabled across eight models, while 17% remain unchanged and up to 87% show no improvement for at least one model.Reported degradation ranges from 7% for Opus 4.6 to 25% for Qwen3-30B.
- Motivation: Skill failures reflect both model disregard of guidance and mismatches between skill assumptions and actual model capabilities.Current skill formats can also increase tokens by 451% without changing pass rates and add execution latency.
- SkVM: SkVM treats skills as code and LLMs as processors, compiling skills for different models while providing unified loading, parsing, and concurrent execution.Its design combines interpreted execution with ahead-of-time and just-in-time compilation.
- SkVM: Ahead-of-time compilation extracts primitive capability requirements, environment configuration, and parallelism opportunities before execution.The capability-based compiler characterizes gaps between target models and skill requirements using 26 primitive capabilities.
- SkVM: At runtime, SkVM solidifies frequently used script templates into executable code and adaptively recompiles skills when capability gaps emerge.The runtime parses compiled artifacts and coordinates available resources and tools to schedule execution reliably.
- Results: 15.3% average task-completion improvement accompanies up to 40% lower token consumption and 3.2×–50× wall-clock speedups across eight LLMs and three harnesses.The evaluation covers SkillsBench and representative skill tasks totaling 118 tasks.
2 Skills in the Wild
Skills are widely distributed, structured around tools and procedures, yet their raw-text execution produces model, harness, and environment mismatches that undermine reliability and efficiency.
- Ecosystem Scale and Distribution: Over 118,000 skills span two platforms, but most receive little use and downloads follow a long-tailed distribution.The collection includes 28,990 skills from clawhub.ai and 89,280 from skills.sh; 89% of skills.sh skills have fewer than 86 downloads.
- Skill Taxonomy: Tool-reference and procedural skills comprise 80% of the ecosystem and depend on agents executing prescribed steps rather than generating content.These skills encode analyzable structure such as tool instructions, scripts, and workflows.
- Problems with Current Skill Usage: Skills loaded as raw text carry implicit assumptions about models, harnesses, and environments, creating three mismatch-driven failure modes.The mismatch space is combinatorial because model, harness, and environment dimensions can vary independently.
- Model Mismatch: 15% of tasks degrade with skills, 17% remain unchanged, and 87% show no improvement for at least one model.A case study reports a weaker model scoring 95 without a skill but repeatedly issuing wrong commands with it.
- Environment Mismatch: Missing dependencies reduce Qwen-model success to 33–67% while increasing failed-workaround output by 2–4×.Even Claude Opus 4.6 generates 56–69% more output tokens when diagnosing missing dependencies.
3 Design Overview
SkVM combines install-time compilation with runtime optimization to adapt skills across heterogeneous models, harnesses, and execution environments.
- Architecture: SkVM combines an AOT compiler that specializes skills at installation with a runtime that resolves execution-time uncertainty.The architecture is organized around compile-time specialization and runtime adaptation.
- AOT Compiler: The AOT compiler performs capability-based compilation, environment binding, and concurrency extraction to produce optimized skill variants.These passes address model and harness mismatch, materialize dependencies, and map latent workflow parallelism to the target harness.
- Runtime & JIT Optimizer: The runtime selects the variant for the current model–harness pair, monitors outcomes, recompiles when capability gaps emerge, and solidifies fixed code patterns.A resource-aware scheduler throttles or suspends concurrent sub-agents when demand exceeds capacity.
4 Skill Compilation
SkVM compiles each skill for a target model, harness, and environment by recovering capability requirements, measuring target proficiency, and applying targeted transformations. It also binds dependencies and extracts executable parallelism from workflow structure.
- SkVM compiles a skill before execution into variants tailored to the target model, harness, and host environment through three sequential passes.The passes address capability mismatch, environment mismatch, and latent workflow parallelism.
- Capability-based compilation: Primitive capabilities provide a model-independent vocabulary for skill requirements, with 26 capabilities across four categories covering 95% of skills.Capabilities are composable, general, and semantically focused on structural correctness.
- Capability-based compilation: The compiler compares extracted skill requirements with cached microbenchmark profiles of the target and selects transforms based on each capability-level gap.Profiling measures proficiency levels for each primitive capability and is reused for compilations targeting the same model and harness.
- Capability-aware transforms: Compensation lowers a requirement through examples, explicit instructions, or stronger constraints, while substitution switches to an alternative implementation when compensation cannot bridge the gap.Compensation is preferred because it preserves the skill’s original intent; substitution can replace a Python workflow with an SQL-based path.
- Capability-aware transforms: For a PPTX skill, SkVM compensated for weak relative-path execution by injecting absolute path resolution instead of switching from python-pptx to PptxGenJS.The target matched four capabilities directly and supported tool.exec at L1 versus the skill’s L2 requirement.
- Concurrency extraction: Concurrency extraction builds a workflow DAG from step inputs and outputs, analyzes internal sub-operations, and maps viable parallel opportunities to harness primitives.Independent tool calls can be grouped into one parallel stage, while unavailable primitives fall back to sequential execution.
5 Skill Runtime with JIT Optimization
SkVM combines ahead-of-time target-specific variants with runtime optimization for uncertainty and repetition that emerge during execution. JIT mechanisms adapt skills from failure evidence and replace repeatedly generated code with validated executable functions.
- AOT compilation creates one skill variant per model–harness target, which the runtime loads with modest one-time compilation overhead.The runtime hides the multiplicity of variants from the agent.
- JIT optimization addresses execution-time uncertainty and recurring code patterns through adaptive recompilation and code solidification.These mechanisms complement AOT specialization by using runtime information.
- Adaptive recompilation: Adaptive recompilation uses repeated failure evidence and self-recovery traces to distinguish task-specific failures from systematic capability gaps.When recompilation performs worse, SkVM rolls back and starts future recompilations from the best observed variant.
- Code solidification: Code solidification identifies parameterized code patterns, validates them across repeated invocations, and promotes matching templates into executable functions.Candidates contain relevance keywords, code signatures, templates, and parameter schemas.
- Code solidification: After promotion, subsequent invocations extract parameters and call the instantiated function without LLM inference, while failures re-enable LLM-based code generation.This provides a correctness fallback for solidified code execution.
- Resource-aware scheduling: The runtime dynamically schedules parallel tasks using API latency, rate-limit signals, CPU usage, memory usage, and other shared-resource conditions.This connects compile-time parallelism opportunities with changing runtime capacity.
6 Evaluation
The evaluation spans diverse tasks, models, baselines, and harnesses, measuring completion, token use, and latency. Figures compare skill variants and staged optimization across model tiers, harnesses, and skill categories.
- Benchmark and methodology: The benchmark covers code generation, data analysis, document creation, and system administration using SkillsBench, PinchBench, and skills from three mainstream sources.Evaluation combines static analysis with LLM-based assessment against predefined criteria.
- Models and baselines: SkVM is evaluated across eight models spanning SOTA, mid-tier, and small capability tiers.The models range from claude-opus-4.6 and deepseek-v3.2 to qwen3-30b and devstral-small.
- Models and baselines: The comparison includes No Skill, Original, and Skill-Creator baselines alongside SkVM-optimized skills.Skill-Creator uses claude-opus-4.6 for optimization.
- Harnesses: Three harnesses—BareAgent, OpenCode, and OpenClaw—provide distinct execution environments for evaluating cross-harness behavior.They differ in abstraction layers, tool dispatch, and integrated functionality.
- Methodology: Each task uses five diverse input instances, and evaluation measures average completion rate, token consumption, and end-to-end execution latency.SkVM first profiles primitive capabilities, then applies AOT and JIT compilation to generate target-specific variants.
- Evaluation views: Figures 10 and 11 compare four skill variants across eight models and three harnesses, and six optimization stages across 14 skill categories.Figure 10 shades model tiers; Figure 11 includes original, AOT-compiled, and three JIT rounds.
6.2 Skill Compilation Effectiveness
SkVM improves completion across models and harnesses, with larger gains for weaker models and further improvements from staged JIT optimization. It also reduces cross-harness variation, though JIT can occasionally regress.
- SkVM-optimized skills achieve the highest average score on every model–harness combination.The comparison includes No Skill, Original, Skill-Creator, and SkVM-Optimized variants.
- Cross-model and harness effectiveness: 25% and 10% improvement over Skill-Creator occur for qwen3-30b and devstral-small, respectively, on BareAgent.The improvement over Skill-Creator grows as model capability decreases.
- Cross-model and harness effectiveness: 4.5% of tasks regress with SkVM-compiled skills versus 15% with original skills, while the cross-harness score gap falls from up to 13 points to at most 5 points.These comparisons are reported for the same evaluation setting across models and harnesses.
- Staged optimization: In 11 of 14 categories, original skills underperform no skill; AOT compilation then raises the average task score by 88%.The staged breakdown is measured on Qwen3-30B with BareAgent.
- Staged optimization: After one JIT round, 8 of 14 skills reach full scores, increasing to 10 of 14 after three rounds.JIT exposes additional capability defects not exposed during AOT compilation.
- Staged optimization: JIT can introduce regressions when an optimization artifact, such as an overly long example file, triggers parsing errors.In the Line Chart task, three revision rounds guided by prior failure logs eventually fixed the issue.
- Efficiency: Optimization points generally combine higher task quality with lower token usage relative to the Original baseline.Figure 12 plots each model–harness pair by score gain and token savings.
6.4 Token and Cost Efficiency with Compilation
SkVM compilation generally improves task success while reducing token consumption, though weaker models can incur modest token increases when improved completion produces longer execution traces.
- For most model–harness combinations, compilation simultaneously improves task success and reduces token consumption.
- Nearly 40% token savings occur for the DS-v3.2 + BareAgent pairing.
- JIT compilation saves tokens by avoiding environment- and tool-error retries during agent execution.
- Weaker models may consume additional tokens because higher completion rates produce longer agent execution traces.
- Environment binding restores performance to complete-environment levels across average score, token use, and duration for all three models.
6.5 Overhead of Target Profiling
Target profiling is a one-time, cached compilation cost, while environment binding addresses missing dependencies and parallelism extraction improves execution efficiency.
- Target profiling: Capability profiling is performed once per model–harness pair and cached for subsequent skill compilations.
- Target profiling: Full capability profiling takes 7.3 minutes for devstral-small and 31.1 minutes for qwen3-30b.The corresponding costs are $0.033 and $0.079.
- Environment binding: Environment binding shifts dependency installation from LLM execution to preexecution preprocessing.This allows the LLM to focus on skill logic.
- Environment binding: Environment binding fully restores execution correctness and substantially reduces token consumption when dependencies are missing.Stronger models can recover autonomously at higher token cost, whereas weaker models may fail.
- Concurrency extraction: SkVM evaluates sequential, DLP, ILP, and TLP execution across eight tasks.
- Concurrency extraction: Parallelism extraction achieves up to 3.2× end-to-end speedup, with TLP producing the largest average improvements.ILP reduces execution time and LLM invocations, while DLP gains scale with data-parallelism degree.
6.8 Evaluating Code Solidification
Code solidification converts reusable generated-code patterns into executable code, producing large latency reductions while retaining a fallback when solidified execution fails.
- For PDF tasks, matching generated-code signatures allow SkVM to instantiate solidified code from templates and input parameters.
- PDF-extract tasks reduce execution time from 10,469–15,116 ms to 206–568 ms, achieving a 19–50× speedup.
- Weather-current reduces execution time from 9,000 ms to 2,000 ms, yielding a 5× improvement bounded by network latency.
- Weather-forecast does not promote because generated code diverges from the predicted signature, so all eight invocations retain LLM generation.
- SkVM falls back to LLM code generation if solidified execution causes task failure.
7 Discussion
SkVM’s discussion identifies nondeterministic natural-language compilation, incomplete capability coverage, and AOT compilation cost as important boundaries of the system.
- Natural-language skill compilation is inherently nondeterministic, although optimization passes and rollback mechanisms stabilize performance improvements across most downstream tasks.
- The current catalog contains 26 primitive capabilities covering the 15,063 analyzed skills.The authors describe iterative expansion as the skill library grows.
- AOT compilation incurs token costs because it invokes an LLM, but repeated execution amortizes the overhead.Compiled skills can also be shared across users and applications.
8 Conclusion
After analyzing over 100,000 skills, the paper presents skill compilation as a systematic path toward portable components across diversifying models and agent harnesses.
- SkVM uses AOT and JIT compilation to optimize skill structure, extract environment dependencies, and identify parallelism opportunities.