Source-linked AI summary
LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin
TL;DR
Textual skills impose repeated context costs and expose procedures, while parametric alternatives can make skills difficult to update or combine. LatentSkill compiles skills into modular LoRA adapters and improves performance across ALFWorld and Search-QA while substantially reducing token overhead.
Problem
Textual skills incur repeated context costs and plaintext exposure, while parametric alternatives fuse skills into parameters, limiting their update, removal, or combination.
Method
LatentSkill uses a pretrained hypernetwork to convert textual skills into modular LoRA adapters mounted on a frozen backbone instead of inserting skill documents into prompts.
Results
LatentSkill outperforms direct in-context skill prompting across ALFWorld and Search-QA while reducing token costs, including 21.4 and 13.4 percentage-point ALFWorld gains.
Takeaways & Limitations
Latent skill weights provide an efficient, modular, controllable, and less directly exposed substrate, with structured geometry and composability when components are aligned.
Abstract
from arXiv · showhide
Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 63.9% fewer prefill tokens on average, and improves Search-QA exact match by 3.0 points while using 71.8% fewer tokens per step. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be continuously modulated via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.
Introduction
LatentSkill converts textual agent skills into modular LoRA adapters generated by a hypernetwork, moving skill knowledge from prompt context into model weight space. This reduces prompt overhead while enabling structured analysis of skill representations, scaling control, and composition.
- Motivation: LLM agents use external textual skills to encode reusable task strategies, tool-use patterns, and recovery heuristics for specialized and long-horizon tasks.
- Motivation: Readable skill prompts consume context, increase prefill cost, and can expose proprietary procedures or mix instructions with untrusted observations.
- Method: LatentSkill uses a trained hypernetwork to generate a skill-specific LoRA adapter from a skill description in one forward pass, then mounts it on the backbone LLM.
- Results: Across ALFWorld and Search-QA, LatentSkill improves over direct in-context skill prompting while substantially reducing prompt overhead from skill text.
- Skill geometry: Generated skill LoRAs exhibit domain-level structure, with skills from different domains forming separable clusters in weight space.
Related Work
Prior work extends LLM agents with externally stored experiential knowledge and natural-language skills injected into context, while hypernetworks generate LoRA adapters for LLMs without per-task fine-tuning. Text-to-LoRA further conditions a shared hypernetwork on task, module, and layer representations, but does not explicitly model interactions among target layers.
- Agent Skills: LLM agents increasingly use external experiential knowledge, including raw trajectories, reflective summaries, hierarchical skills, and reusable strategic principles.These approaches inject knowledge at decision time to extend agents beyond a single model’s knowledge boundary.
- Agent Skills: Mainstream agent harnesses adopt natural-language skills that are injected into the context window.The passage names Anthropic’s Agent Skills specification and adoption by Claude Code, Cursor, and Gemini CLI.
- Hypernetworks and LoRA: Hypernetworks generate target-network weights and can produce LoRA adapters in a single forward pass, bypassing iterative per-task fine-tuning.This provides a learned-network mechanism for generating LLM adapters.
- Hypernetworks and LoRA: Text-to-LoRA uses a shared MLP hypernetwork conditioned on task, module, and layer representations to generate LoRA weights, but does not explicitly model interactions among target layers.The method is presented as a recent application of hypernetworks to LLM adaptation.
Method
LatentSkill compiles textual skills into LoRA adapters generated by a skill compiler, replacing per-step textual conditioning with parameter conditioning on a frozen LLM. The compiler is pretrained on document-level reconstruction and completion tasks, fine-tuned on teacher trajectories, and supports cached, strength-controlled, and composable adapters at inference.
- Latent skill formulation: LatentSkill converts each textual skill s into a generated LoRA update Δs = Gϕ(s), which is mounted on the frozen backbone instead of inserted into the prompt.The adapted model then predicts from task history alone, with LoRA augmentation controlled by an injection coefficient α.
- Document-level pretraining: The compiler is pretrained with a 1:1 mixture of document reconstruction and truncated-prefix completion objectives while the backbone remains frozen.Information useful for predicting the target must pass through the generated adapter because the skill document is provided to the compiler rather than directly to the adapted backbone.
- Trajectory-supervised fine-tuning: Trajectory-supervised fine-tuning generates one latent skill per skill document and shares that adapter across every decision step in the teacher trajectory.Only compiler parameters ϕ are updated, encouraging trajectory-consistent skill effects rather than per-step adaptations.
- Inference-time control: At inference, skills can be compiled once into an adapter cache, omitted from prompts, and mounted with coefficient αk, where αk = 0 recovers the frozen backbone.Larger αk values increase the latent skill’s influence during prediction from the current history.
- Inference-time composition: Multiple selected skills are composed in weight space, while component-level composition adds retained shared and skill-specific components with optional coefficients γc.Component decomposition is designed to avoid over-amplifying common behavior when skills share subcomponents.
Experiments
Across ALFWorld and Search-QA, LatentSkill delivers the strongest reported performance while substantially reducing token costs versus in-context skills. Additional experiments show gains in out-of-domain generalization, controllable skill injection, aligned LoRA composition, and robustness to perturbations and prompt attacks.
- Method Performance: LatentSkill achieves 74.3% and 69.4% average success on ALFWorld seen and unseen splits, exceeding In-Context Skill by 21.4 and 13.4 points.On Search-QA, it achieves 35.6% average EM, outperforming RAG by 1.2 points and SHINE by 1.5 points.
- Out-of-Domain Generalization: 23.9 WebShop score for the SFT checkpoint and 18.5 for the pretrained checkpoint exceed Vanilla’s 10.3 and In-Context Skill’s 6.7.WebShop trajectories were excluded from training, and the in-context result suggests long skill prompts may dilute attention to current states and admissible actions.
- Ablations: 20.7 and 29.3 points higher ALFWorld seen performance than Full SFT and Shared LoRA, respectively, show teacher trajectories alone do not explain LatentSkill’s gains.LatentSkill also exceeds Per-skill LoRA by 28.6, 19.4, and 2.8 points on ALFWorld seen, ALFWorld unseen, and Search-QA.
- Token Efficiency: 63.6% and 64.2% lower ALFWorld prefill costs on seen and unseen splits accompany 21.4- and 13.4-point success gains over In-Context Skill.Search-QA per-step token cost falls 71.8% while EM improves by 3.0 points; latent skills also reduce seen-split trajectory length from 35.0 to 28.4 steps.
- Continuous Skill Control: 74.29% seen performance at α=0.6 and 70.90% unseen performance at α=0.5 peak before falling to 22.86% and 8.21% at α=1.2.Optimal coefficients vary across tasks, with weaker-backbone tasks such as Pick2 and Clean preferring stronger injection.
- Skill Composition: 84.6% seen and 77.8% unseen success make Component Merging the best composition strategy, adding three seen and one unseen success over Look-Only without losing originals.The results indicate that reliable LoRA composition requires semantically aligned components.
Conclusion
LatentSkill moves reusable procedural knowledge from textual context into modular LoRA adapters generated by a pretrained hypernetwork, improving performance while reducing repeated prefill overhead. Its generated skill weights exhibit structured semantics, controllability, composability under alignment, and reduced plaintext exposure.
- Core contribution: LatentSkill converts textual agent skills into modular LoRA adapters through a pretrained hypernetwork, moving reusable procedural knowledge from context space into weight space.This design supports modular skill loading without repeatedly injecting skill text into the prompt.
- Empirical outcome: Across ALFWorld and Search-QA, LatentSkill improves over direct in-context skill prompting while substantially reducing repeated prefill overhead.The efficiency gain comes from removing repeated skill-text injection during agent operation.
- Skill-weight properties: Generated skill LoRAs form a structured semantic geometry, can be controlled through the injection coefficient, and can be composed in parameter space when skill components are properly aligned.These analyses identify semantic structure, continuous control, and a condition for parameter-space composition.
- Implications: Latent skill weights provide a practical substrate for LLM agents whose skills are efficient, modular, controllable, and less directly exposed as plaintext prompts.This summarizes the paper’s proposed benefits of storing skills in weights rather than prompt context.