SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

1 INTRODUCTION

• Latent reasoning avoids exchanging intermediate natural-language text, but inserting textual skills into prompts requires repeated prefilling that can erode its inference-speed gains.KV prefixes and skill-specific LoRA adapters offer alternatives, but their storage can grow with prefix length and layer count or with the skill library.Source: §1 · pp. 1–2
• SkillMem stores each reusable skill as a small external latent memory, retrieved for a task and supplied during latent multi-agent reasoning without adding its textual description to each prompt.This separates skill selection from how an agent uses the skill.Source: §1 · p. 2
• A role-conditioned reader lets agents share one retrieved memory while producing role-dependent updates to their hidden states.The same procedure can serve agents such as planners and critics at different stages and for different purposes.Source: §1 · p. 2
• Behavioral distillation trains the memory and shared reader to reproduce the token-prediction changes induced by a textual skill while keeping the backbone frozen.Experiments use textual skills derived from paired successful and failed multi-agent trajectories, though existing textual skills can also be compiled directly.Source: §1 · p. 2
• Across six benchmarks, SkillMem improves on the latent MAS baseline; Qwen3-4B gains include 17.97% to 35.63% on GPQA-Diamond and a 4.23× end-to-end speedup over TextMAS.Each skill requires 9.6 KB of skill-specific storage; for 100 skills, total storage is 38.3× smaller than KV Skill and 86.6× smaller than LoRA Skill.Source: §1 · p. 2, Figure 1 · p. 2

Latent multi-agent reasoning and memory

• LatentMAS uses continuous states for agent reasoning and communication; KVComm and RecursiveMAS (Yang et al., 2026) study efficient communication and computation.LatentMem (Fu et al., 2026) builds agent-specific memories from retrieved interactions, while Agent Primitives (Jin et al., 2026) represents predefined operations with KV states.Source: §2 · p. 2

Skill representations and behavioral distillation

• S2L (Zhang & Qi, 2026) and LatentSkill (Yu et al., 2026) compile skills into LoRA adaptations, while SkillMem stores skill effects in independently retrievable external memory.SEED (Wu et al., 2026) distills skill-induced behavioral changes into the agent policy; SkillMem uses a reader shared across skills and roles with a frozen backbone.Source: §2 · p. 3

Continuous representations and context compression

• Prompt and prefix tuning, Gisting, and AutoCompressor motivate continuous conditioning or context compression.SkillMem instead learns a separate memory for each reusable procedure and accesses it through a shared, role-conditioned reader.Source: §2 · p. 3

Experience-based skill learning and routing

• Voyager and ExpeL reuse trajectory knowledge, while SkillRL (Xia et al., 2026) builds skills from single-agent trajectories and jointly updates skills and policy.SkillMem derives textual skills by comparing successful and failed multi-agent trajectories; its retriever selects a task skill, while the method focuses on storing and using it across agents.Source: §2 · p. 3

3 PRELIMINARIES: LATENT MULTI-AGENT REASONING

• During autoregressive generation, the decoder-only Transformer retains previously computed KV states to avoid recomputing earlier positions.Latent reasoning uses continuous vectors in place of some intermediate natural-language tokens.Source: §3 · p. 3
• A mapping ϕ transforms the final-layer hidden state at position t into zt+1, the input embedding for the next position, allowing reasoning to continue without decoding and re-embedding an intermediate word.The continuous representation replaces an intermediate natural-language token.Source: §3 · p. 3
• In latent multi-agent reasoning, role-specialized agents pass layer-wise KV caches containing prior latent context and their added states, and only the final agent generates a natural-language answer.Agents may act as planner, critic, refiner, or final-answer agent.Source: §3 · p. 3

4 METHOD

• Given a task query, SkillMem retrieves a skill and makes its memory available to every agent while agents continue communicating through KV states.Skills may be provided directly or discovered from prior multi-agent trajectories.Source: §4 · p. 3
• A role-conditioned reader uses the retrieved memory to update agents' hidden states according to their roles.Source: §4 · p. 3
• SkillMem trains memories and the reader to reproduce how textual skills change predictions from the frozen backbone.The experiments use discovered skills, while direct skill provision is also supported.Source: §4 · p. 3

1. Skill Discovery 2. Skill Compilation 3. Inference with Latent Skill

• SkillMem compiles reusable textual skills into latent memories that are retrieved for multi-agent reasoning and shared across agents while inter-agent communication remains latent.The overview describes discovery from paired multi-agent trajectories, compilation by matching behavioral effects, and inference with a shared retrieved memory.Source: Figure 2 · p. 4

4.1 SKILL MEMORY AND SKILL RETRIEVAL

• A task query retrieves the library skill with the highest cosine similarity to its encoded representation, and each skill is stored as an external memory.Memory storage scales as O(Mdmem), independent of the textual skill length Ts, while cached KV storage scales as O(LTsdkv).Source: §4.1 · p. 4

4.2 ROLE-CONDITIONED MEMORY READER

• A shared role-conditioned reader uses each agent’s role and current reasoning state to read the memory and update hidden states, without separate skill storage for each role.The reader’s parameters are shared across skills and roles at each insertion layer, while agents continue to communicate through latent states.Source: §4.2 · pp. 4–5

4.3 BEHAVIORAL SKILL DISTILLATION

• Behavioral distillation trains the memory and reader to reproduce the textual skill’s effect on token predictions while keeping the backbone frozen.The procedure scores the same teacher response with the skill text, without it, and with the memory; training also preserves base behavior where the skill provides limited support.Source: §4.3 · pp. 5–6

4.4 SKILL DISCOVERY

• Experimental skills are extracted by comparing matched successful and failed multi-agent trajectories for the same task, then removing instance-specific details and deduplicating candidates.Skill discovery is not required for compilation: existing textual skills can be compiled directly, while explored zero-training trajectory-to-memory variants were substantially less reliable.Source: §4.4 · pp. 6–7

• SkillMem transfers text-supervised skills by matching their behavioral effect on the frozen backbone, rather than reconstructing skill wording.Source: §5 · p. 7
• Input-output alignment bounds the hidden-state mismatch reaching the SkillMem reader when the insertion-layer mapping is locally Lipschitz.Source: §5 · p. 7
• The transfer argument models textual-skill and SkillMem behavioral effects at a reader state and assumes both are locally Lipschitz.Source: §5 · p. 7
• The transfer bound separates behavioral-distillation error from the representation gap introduced by latent recurrence; input-output alignment reduces the latter.Source: §5 · p. 7

6.1 EXPERIMENTAL SETUP

• Evaluation uses frozen Qwen3-4B/8B/14B across six benchmarks, disjoint calibration examples, three seeds, eight-vector memories, and TextMAS, LatentMAS, KV, Soft Prompt, and LoRA baselines.The same skill library and retrieved skill are used across skill-based methods.Source: §6.1 · pp. 7–8

6.2 MAIN RESULTS

• SkillMem beats LatentMAS and all three alternative skill representations on every Qwen3-4B benchmark, leads on five of six tasks, wins three named tasks on 8B, and has no consistent advantage on 14B.The 14B baselines already exceed 90% accuracy on GSM8K and ARC, leaving less room for additional skill conditioning.Source: §6.2 · p. 8, Table 1 · p. 8

6.3 STORAGE AND EFFICIENCY

• SkillMem uses 9.6 KB per skill plus one shared reader, versus 13.0 MB for KV Skill and 30.7 MB for LoRA Skill; its storage advantage ranges from 3.9× to 308× over KV Skill.The scaling comparison spans libraries of 10 to 1000 skills.Source: Table 2 · p. 8, §6.3 · p. 9
• SkillMem's end-to-end latency is 7.95 s, versus 8.18 s for LatentMAS, 12.98 s for Text Skill, and 12.01 s for KV Skill; training 10 MedQA skills takes 11.7 minutes versus 110 minutes for separate LoRA adapters.The fixed eight-vector memory adds neither skill tokens nor skill-specific KV prefixes; shared-reader training is 9.4× faster than separate adapters.Source: Table 2 · p. 8, §6.3 · p. 9

6.4 ABLATION STUDIES

• Removing role conditioning drops average accuracy from 75.15% to 67.37% on Qwen3-4B and from 77.97% to 76.50% on Qwen3-8B; per-role memories offer no advantage over one shared memory.The ablation compares reader variants while holding retrieved skills and evaluation examples fixed.Source: §6.4 · p. 9, Table 3 · p. 9
• M=8 yields the best average accuracy at 64.08%, while M=2 reaches 63.63% with one quarter as many vectors; increasing capacity to M=16 reduces accuracy.The observed relationship between memory size and accuracy is non-monotonic.Source: §6.4 · p. 9, Table 3 · p. 9
• Removing any distillation loss lowers macro-average accuracy from 64.55% to 62.43%–62.93%, with the largest drop without Lshift; the full objective performs best overall.The results support direct matching of the behavioral change induced by the textual skill, with the three losses providing complementary signals.Source: §6.4 · p. 9, Table 3 · p. 9

7 CONCLUSION AND FUTURE WORK

• SkillMem stores reusable skills as compact external memories, with a role-conditioned reader and behavioral distillation transferring textual-skill effects without modifying the frozen backbone.The reported scope covers six benchmarks and three model scales, with low skill-specific storage and efficient latent inference.Source: §7 · p. 9
• The same memory can guide agents in different roles through role-conditioned access, rather than requiring a separate skill representation for each role.Behavioral distillation transfers the effect of a textual skill while the backbone remains frozen.Source: §7 · p. 9
• Future work includes retrieving and composing multiple skills in one reasoning process, resolving conflicting guidance, conditioning retrieval on role and reasoning state, and updating memories online.Source: §7 · p. 10

AI ASSISTANCE DISCLOSURE

• The authors used language models for editing and implementing parts of the code, which they reviewed, tested, and verified; they attribute the research and conclusions to the authors.The authors state that they developed and validated the research ideas, methodology, experimental design, analysis, and scientific conclusions.Source: AI ASSISTANCE DISCLOSURE · p. 10

A SKILL DISCOVERY

• For each benchmark, discovery uses 20 calibration examples and matched successful and failed multi-agent trajectories from the same tasks.The trajectories preserve planning, critique, refinement, and verification stages; the backbone compares each pair to identify reusable procedures.Source: A SKILL DISCOVERY · p. 12
• Candidate skills specify a name, when the procedure is useful, and the procedure itself; instance-specific details are removed and overlapping candidates are deduplicated.The resulting libraries are discovered independently at each backbone scale and can therefore differ across scales.Source: A SKILL DISCOVERY · pp. 12–13

A.1 DISCOVERED SKILLS

• The Qwen3-4B appendix lists discovered skills by benchmark, while the main discovery table reports skill counts by benchmark and model scale.The listed skills include procedures such as restating a target quantity and normalizing units before combining measurements.Source: Table 5 · p. 13, Table 4 · p. 13, §A.1 · p. 14

B SKILL RETRIEVAL

• At inference, the system retrieves the top-1 skill by cosine similarity between query and skill embeddings, then assigns that same skill to every agent in the rollout.Routing is performed once per example, with no minimum similarity threshold; cached routing decisions are reused across skill-based methods for matched comparisons.Source: B SKILL RETRIEVAL · p. 15

C EVALUATION PROTOCOL

• Evaluation uses temperature 0.6 and top-p 0.95, with eight latent reasoning steps per agent and a maximum of 512 generated tokens for the final answer.Reported accuracy averages three evaluation seeds that vary only the decoding RNG on a fixed trained checkpoint.Source: C EVALUATION PROTOCOL · pp. 15–16

D.1 SKILLMEM HYPERPARAMETERS

• The role embedding is computed from frozen backbone representations of each role instruction, while each insertion layer has its own reader weights and shares the skill memory bank.The role embedding is not learned, and the reader modules are independently parameterized across layers.Source: §D.1 · p. 16

D.2 INTERMEDIATE-ROLE BEHAVIORAL DISTILLATION

• For intermediate-role distillation, the authors cache one greedy teacher continuation per skill-role pair and calibration example, generated from that role’s prompt without other-agent context.Teacher tokens provide fixed teacher-forcing targets during training and are not part of the deployed inference pipeline; only the judger’s continuation is correctness-filtered.Source: §D.2 · p. 17

D.3 SKILL-TO-LORA (S2L) BASELINE HYPERPARAMETERS

• The Skill-to-LoRA baseline trains one independent adapter per discovered skill using teacher-forcing supervision from all four roles.Its setup is the role-agnostic analogue because the LoRA baseline has no native role-conditioning mechanism.Source: §D.3 · p. 17

E DETAILED ANALYSIS OF TEXT-TO-LATENT SKILL TRANSFER

• The transfer analysis separates text-to-latent transfer into input-space alignment and propagation through the frozen backbone to the SkillMem reader’s hidden states.The reader consumes intermediate hidden states rather than directly reading the final-layer latent state.Source: E DETAILED ANALYSIS OF TEXT-TO-LATENT… · p. 17, §E.2 · p. 18

E.1 INPUT-OUTPUT ALIGNMENT IN LATENT REASONING

• LatentMAS maps final-layer hidden states toward the input space because directly reusing them as inputs may introduce distribution mismatch.The remaining discrepancy after alignment is represented as an input mismatch; the text-derived and aligned latent inputs need not be identical.Source: §E.1 · p. 18

E.2 FROM INPUT ALIGNMENT TO READER-STATE ALIGNMENT

• If the frozen input-to-insertion-layer mapping is locally Lipschitz, input alignment bounds the mismatch between the text-derived and latent reader states.The states are the reader inputs induced by text-derived and latent reasoning, respectively.Source: §E.2 · p. 18

E.3 BEHAVIORAL EFFECT MATCHING

• Behavioral distillation matches the textual skill’s prediction change with the memory-induced change on text-derived training states, allowing for residual distillation error.The analysis assumes both behavioral-effect functions are locally Lipschitz in the reader-state region.Source: §E.3 · pp. 18–19

E.4 PROOF OF TEXT-TO-LATENT BEHAVIORAL TRANSFER

• The proof establishes Proposition 1 by combining the behavioral-effect and reader-state alignment bounds.The transfer bound carries a residual distillation error and an additional term tied to latent-input mismatch and local sensitivity.Source: §E.4 · p. 19
• The bound’s δ term measures distillation error on training states, while (LF + LG)LΦϵalign captures additional error when applying the skill to latent states.LatentMAS targets the input component through its alignment operator, and the frozen backbone maps text-derived and aligned latent inputs to the reader interface.Source: §E.4 · p. 19
• The analysis does not require identical text and latent states, but assumes the aligned trajectory stays in a region where the backbone and skill-effect functions vary smoothly.This local smoothness condition bounds transfer only within the region covered by the analysis.Source: §E.4 · p. 19

F DIRECTLY DISCOVER LATENT VECTOR AS SKILLS

• Contrastive extraction averages success-minus-failure hidden-state representations from matched attempt pairs, while latent generation projects the model’s own continuous reflection steps into the skill-vector dimension.Both variants use fixed random projections and rescale their vectors; contrastive extraction also rescales its much smaller raw norm to match trained skills.Source: F DIRECTLY DISCOVER LATENT VECTOR AS… · pp. 19–20
• Contrastive extraction matches or marginally beats the no-skill baseline on MedQA and GSM8K, but collapses into degenerate repetition on GPQA-D across all three seeds.Table 9 reports three-seed accuracy deltas against no-skill LatentMAS on Qwen3-8B, with each discovered skill forced on every held-out example.Source: Table 9 · p. 20, F DIRECTLY DISCOVER LATENT VECTOR AS… · p. 20
• Neither zero-training variant is competitive with text-skill distillation, and latent generation never exceeds the no-skill baseline.The shared reader is trained on a particular distribution of skill memories, so untrained off-distribution directions can be safe on some benchmarks and destructive on others without an obvious advance predictor.Source: F DIRECTLY DISCOVER LATENT VECTOR AS… · p. 20