Source-linked AI summary
Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
Taeil Kim, Kangsan Kim, Sung Ju Hwang
TL;DR
Small agents have limited successful experiences for memory systems, leaving teacher-to-student memory transfer underexplored. AMD addresses this with hierarchical memories from teacher trajectories and consistently improves accuracy across three tool-use benchmarks, including a 27.2%p average gain on AppWorld.
Problem
Memory’s potential for small agents remains underexplored because their repositories contain few successful trajectories and many failures.
Method
AMD transfers teacher experiences through hierarchical Workflow, Subtask, and Function memories spanning task, subtask, and function granularities.
Results
AMD consistently outperforms zero-shot and existing memory-based baselines across three benchmarks and four students, gaining 27.2%p average accuracy on AppWorld.
Takeaways & Limitations
AMD provides a foundation for transferring stronger agents’ accumulated experiences to small agents through structured memory.
Takeaways & Limitations
AMD is evaluated only on text-based tool-use benchmarks, and its offline frozen memory cannot adapt to students’ test-time experiences or distribution shifts.
Abstract
from arXiv · showhide
Memory systems have shown promise for improving agent performance, but their potential remains largely unexplored for small language models, which struggle to generate sufficient successful trajectories on their own. We propose Agent Memory Distillation (AMD), a training-free framework that transfers structured knowledge from a large teacher agent to a small student agent through hierarchical memory. AMD constructs three complementary memory types from successful teacher trajectories: Workflow memory encodes task-level strategies, Subtask memory provides concrete behavioral examples at an intermediate granularity, and Function memory captures per-function calling conventions and common pitfalls. Workflow and Subtask memories are injected proactively at the start of each task, while Function memory is retrieved reactively upon tool-calling errors. We evaluate AMD on three tool-use benchmarks using four student models (4B-8B parameters) with GPT-5-mini as the teacher, achieving average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V3, and ToolSandbox, while consistently outperforming existing memory-based baselines. Further analysis shows that Subtask memory contributes the largest gains, teacher effectiveness depends on both teacher capability and student compatibility, and 4B-sized students benefit most from AMD.
1 Introduction
AMD addresses the difficulty of improving small agents with memory by transferring teacher experience through a hierarchy spanning task, subtask, and function levels. Across three benchmarks and four small student models, AMD substantially improves accuracy over zero-shot performance.
- Motivation and challenge: Small agents have limited memory potential because low task success rates produce repositories dominated by failures and few successful experiences.This scarcity of successful trajectories limits the benefits of memory utilization despite possible reflection on failures.
- Motivation and challenge: Naive transfer of teacher memories yields only marginal improvements because the teacher–student capability gap hinders effective memory distillation.The passage compares naive transfer with student-generated memory and relates the obstacle to knowledge gaps in conventional distillation.
- Agent Memory Distillation: AMD constructs hierarchical teacher memories across task-level, subtask-level, and function-level granularities to make teacher knowledge more comprehensible and applicable to small students.Workflow memory captures overall task strategies, while Subtask memory provides concrete action examples for subtasks.
- Evaluation and results: 27.2%p, 11.2%p, and 3.4%p are AMD’s average accuracy gains on AppWorld, BFCL V3, and ToolSandbox, respectively, over the zero-shot baseline.The evaluation uses four student models with 4B or 8B parameters and GPT-5-mini as the teacher.
- Evaluation and results: AMD consistently improves performance across all student models and outperforms memory-based baselines in the reported evaluation.The introduction presents AMD as a systematic investigation of teacher-to-student memory transfer for small agents.
2 Related Work
Prior work has explored memory for agent improvement and knowledge distillation for transferring capabilities from stronger teachers to smaller students. However, existing distillation approaches generally require costly parameter updates or do not let students directly leverage teacher memory.
- Memory-based agents: Memory systems help agents reuse successful patterns, avoid repeated failures, and improve through self-feedback, experience extraction, and retrieval.Reflexion uses verbalized self-feedback and episodic memory buffers, while ExpeL extracts cross-task insights for future inference.
- Knowledge distillation: Knowledge distillation transfers knowledge from stronger teacher models to smaller students and has been effective across diverse LLM settings.Agent Distillation transfers full task-solving behaviors, including retrieval and code tool use, while SCoRe uses student-centered short-horizon reinforcement learning.
- Research gap: Existing distillation methods rely on costly parameter updates or fail to provide direct access to teacher memory.Agent-Distill is training-free but reuses teacher-generated MCPs rather than exposing teacher memory directly.
3 Agent Memory Distillation
Agent Memory Distillation (AMD) transfers structured knowledge from successful large-teacher trajectories to small student agents through a hierarchical memory store. It combines proactive Workflow and Subtask guidance with reactive Function-memory retrieval to guide student reasoning and tool use.
- Memory Retrieval and Utilization: Workflow and Subtask memories are injected at task start, whereas Function memory is retrieved after tool-call errors and appended as corrective hints.Workflow retrieval uses the task instruction, Subtask retrieval uses up to six ordered labels, and Function retrieval uses the failing function name plus similarity ranking.
- Hierarchical Memory Generation: AMD organizes successful teacher experience into Workflow, Subtask, and Function memories at progressively finer levels of task granularity.Workflow captures high-level strategies, Subtask stores intermediate behavioral examples, and Function records individual tool-invocation knowledge.
- Hierarchical Memory Generation: Workflow memory abstracts each successful trajectory into a task-level plan covering tools, preconditions, decision rules, validation cues, and failure patterns.The plan omits dynamic runtime values and enables the student to identify relevant tools and establish an action sequence.
- Hierarchical Memory Generation: Subtask memory segments successful trajectories into semantically meaningful units paired with labels, descriptions, tool-call or executable-code examples, and observations.These examples bridge high-level workflow plans and low-level tool calls.
- Hierarchical Memory Generation: Function memory stores function names, concrete teacher invocations, surrounding context, observations, and optional documentation, indexed by function name for targeted retrieval.It captures fine-grained calling knowledge and can include argument and response schemas.
4 Experiment
AMD is evaluated across three tool-use benchmarks and four small student models against zero-shot, teacher, and memory-based baselines. It consistently improves accuracy, sometimes surpasses the teacher, and reduces inefficient student interactions.
- Experimental Setup: AMD is evaluated on AppWorld, BFCL V3, and ToolSandbox using four student models with GPT-5-mini as teacher.AppWorld uses 168 test_normal tasks, while BFCL V3 evaluates function invocation with accurate arguments.
- Main Results: AMD consistently outperforms all baselines across every evaluated student model and benchmark.Compared methods include zero-shot, the teacher agent, ReasoningBank, MemP, and SASM.
- AMD vs. Baselines: 27.2%p, 11.2%p, and 3.4%p are AMD’s average accuracy gains over zero-shot on AppWorld, BFCL V3, and ToolSandbox, respectively.The gains are averaged across the four student models.
- Interaction Efficiency: AMD reduces inefficient interaction: Qwen3-4B’s AppWorld turn count falls from 23.8 zero-shot turns to 14.9, closer to the teacher’s 10.1.Gemma4-E4B reaches 7.2 turns with AMD on AppWorld.
5 Analysis
The analysis shows that Subtask memory is the most important component, while teacher quality, student size, retrieval count, and representation format govern AMD’s effectiveness. The three memory types operate complementarily across distinct layers of agent competence.
- Memory component analysis: Workflow memory consistently improves zero-shot performance, while adding Subtask memory produces the largest incremental gains, especially on complex AppWorld tasks.On Qwen3-4B, Subtask memory adds +25.0%p over Workflow memory alone on AppWorld.
- Memory component analysis: Function memory provides smaller additional gains when combined with Workflow and Subtask memories, whereas student-generated memories underperform teacher-derived memories.
- Teacher effects: 91.08% teacher accuracy from GPT-5.5 yields 58.93% student accuracy for Qwen3-8B, with lower teacher accuracy generally producing weaker transfer.The reported sequence is GPT-5.5, DeepSeek V4 Pro, GPT-5-mini, and Qwen3-32B, whose teacher accuracies are 91.08%, 81.55%, 50.00%, and 34.42%, respectively.
- Teacher effects: 49.40% accuracy for Qwen3-4B surpasses DeepSeek V4 Pro’s 38.10% despite substantially lower teacher accuracy, indicating that AMD mitigates the capability gap.A modest gap remains between the 4B student and teacher models, leaving room for refinement.
- Student model size: +34.52%p is the peak accuracy gain at 4B, while AMD accuracy rises from 21.43% at 1.7B to 52.68% at 14B and gains diminish for larger students.The corresponding AMD accuracies are 49.40% at 4B and 51.79% at 8B.
- Retrieval count: k=1 is optimal or near-optimal across memory types, while increasing retrieval count generally fails to improve accuracy and reduces Subtask performance from 49.40% to 33.34%.Workflow memory declines moderately, whereas Function memory remains stable as k increases.
- Memory representation: 49.40% versus 44.05% favors natural language for Workflow memory, whereas 49.40% versus 23.21% favors code-centric Subtask memory.The results suggest that high-level planning transfers best through prose, while low-level execution knowledge benefits from concrete executable code.
- Case study: Function memory completes a Venmo task after temporal filtering succeeds but withdrawal fails, supplying the correct API usage pattern for balance extraction.The case study demonstrates that Workflow, Subtask, and Function memories resolve distinct failure modes in a complementary hierarchy.
6 Conclusion
AMD is a training-free framework that transfers teacher-agent experiences to small student agents through hierarchically structured memory. It addresses the limited effectiveness of naive memory transfer by organizing knowledge into three complementary memory types at different task granularities.
- 6 Conclusion: AMD transfers teacher-agent experiences to small student agents through hierarchically structured memory without training.The framework is explicitly described as training-free.
- 6 Conclusion: Naive memory transfer yields only marginal improvements because of the capability gap between teacher and student agents.Simply providing high-quality teacher memories does not bridge this gap.
- 6 Conclusion: AMD constructs three complementary memory types at different levels of task granularity.The supplied passage begins specifying Workflow memory as one of these types.
Limitations · A Algorithmic Details
AMD’s evaluation is limited to text-based tool-use benchmarks, leaving its generalization and adaptive teacher selection unresolved. Its inference pipeline proactively retrieves workflow and subtask memories, then reactively retrieves function records after tool-call errors.
- Limitations: AMD is evaluated only on text-based tool-use benchmarks involving Python APIs or structured function calls.Its generalization to less structured agentic settings remains unverified.
- Limitations: Generalization remains unverified for multimodal environments requiring visual grounding and coding tasks with open-ended generated code.These settings differ from fixed sets of callable operations.
- Limitations: Adaptive teacher selection—determining which teacher best suits a given student—remains an open problem for future work.
- A Algorithmic Details: At task start, the student retrieves the top-k workflow memory entry using the task instruction and prepends the insight to the system prompt.
- A Algorithmic Details: The student decomposes the task instruction into ordered subtask labels, retrieves each independently from subtask memory, and deduplicates retrieved segments.
- A Algorithmic Details: During execution, each tool call is conditioned on the preceding observation, and errors trigger extraction of failing function names for function-memory lookup.
- A Algorithmic Details: Candidate function records are ranked by cosine similarity, with top-k examples formatted as hints and injected only after failures to avoid inflating successful-execution context.
B Additional Experiment Results · B.1 Impact of Each Memory Type · B.2 Detailed Case Studies by Memory Type
The ablations show that Subtask memory contributes the largest gains, while Function memory provides smaller, model-dependent improvements and student-generated memories underperform AMD. Case studies illustrate that workflow, subtask, and function memories correct planning, API sequencing, and response parsing errors through proactive or reactive guidance.
- B.1 Impact of Each Memory Type: Subtask memory provides the largest incremental gain over Workflow memory, while Function memory adds smaller improvements across nearly all models and benchmarks.This contribution ordering holds for almost every evaluated model and benchmark.
- B.1 Impact of Each Memory Type: 30.36% to 27.38%: adding Function memory to LLaMA3.1-8B’s WF+ST configuration lowers AppWorld accuracy.The passage attributes this exception to weaker instruction following and Function memory’s additional error-recovery context causing deviation from planned execution.
- B.1 Impact of Each Memory Type: 30.36% under WF to 40.48% under WF+FN: Gemma4-E4B’s AppWorld accuracy improves when Function memory is added.Function memory consistently improves this model across both benchmarks and all configurations, suggesting stronger incorporation of reactive error-correction guidance.
- B.1 Impact of Each Memory Type: Student Memory performs close to zero-shot on AppWorld and substantially below AMD on both benchmarks.The student’s unsuccessful trajectories lack the structured, reliable knowledge needed for effective memory-guided execution, underscoring teacher-trajectory quality.
- B.2 Detailed Case Studies by Memory Type: Workflow memory supplies upfront identity-resolution procedures, preventing the Qwen3-4B agent from calling a nonexistent API and repeatedly revisiting documentation.Without memory, the agent parses the trip note but lacks the intermediate step of resolving names into contact identities and reaches the step limit.
- B.2 Detailed Case Studies by Memory Type: Subtask memory’s upfront API sequence with pagination changes an incorrect $341 total into the correct $833 total.The memory helps the agent retrieve all pages after the memoryless agent uses the wrong endpoint and retrieves only one page.
- B.2 Detailed Case Studies by Memory Type: Function memory’s on-demand multi-key parser corrects a failed API response interpretation and enables completion of the withdrawal.Without memory, the agent silently skips a $391 withdrawal by using the wrong response key; the teacher-derived parser reads the correct field on the first attempt.
B.3 Robustness of Distilled Memory under Disjoint Evaluation
Disjoint evaluation tests whether AMD’s gains reflect distilled-memory utility rather than overlap between memory construction and evaluation tasks. Across two protocols, Qwen3-4B retains monotonic improvements through WF, WF+ST, and WF+ST+FN, with the full configuration outperforming Zero-shot on every benchmark.
- Motivation: The section separates memory-construction tasks from evaluation tasks to determine whether gains arise from distilled-memory utility rather than task overlap.Both evaluation protocols ensure that gains come only from memory distilled from other tasks.
- Evaluation protocols: Cross-split evaluation uses a 7:3 disjoint split, distilling memory from the construction subset and testing on the held-out evaluation subset.All experiments use Qwen3-4B as the student model.
- Evaluation protocols: Self-excluded retrieval distills memory over the full benchmark but lets each task retrieve memory from other tasks, preserving a larger pool without self-use.This protocol prevents any task from using memory distilled from itself.
- Results: Both protocols preserve monotonic gains from Zero-shot to WF, WF+ST, and WF+ST+FN, and WF+ST+FN outperforms Zero-shot on AppWorld, BFCL V3, and ToolSandbox.Because neither setup permits self-derived memory, the results support gains arising from distilled-memory utility.
B.4 Robustness of Distilled Memory across Repeated Runs
AMD’s improvements remain robust across repeated runs: it outperforms zero-shot on all three benchmarks while exhibiting low variability across five trials.
- Experimental setup: Five repeated runs evaluated zero-shot and AMD with Qwen3-4B under the same setup, reporting mean and standard deviation of success rates.The evaluation covered all three benchmarks.
- Results: AMD achieves higher mean success rates than zero-shot across all three benchmarks, with standard deviations below one percentage point in every setting.The variability is substantially smaller than the corresponding mean improvements, indicating stable gains across runs.
C Additional Implementation Details · D Memory Examples and Generation Prompts
The implementation details specify how AMD memories are generated, retrieved, and applied, while concrete examples and prompts illustrate their hierarchical structure and reuse across tasks. The setup also defines student decomposition, benchmark protocols, and representative workflow, subtask, and function memories.
- C Additional Implementation Details: Workflow memory generation produces a short, retrieval-friendly strategy from successful teacher trajectories, covering tools, preconditions, decision rules, validation cues, and failure patterns.Concrete runtime values are replaced with typed placeholders such as <ID>, <EMAIL>, and <FILE_PATH>.
- C Additional Implementation Details: Subtask memory segments successful trajectories at semantic boundaries into labeled, one-sentence descriptions, using API-call breakpoint hints to guide segmentation.The student decomposes each task into at most six ordered subtask labels before tool calls, then uses those labels as retrieval queries.
- C Additional Implementation Details: Memory retrieval encodes entries with text-embedding-3-small and applies cosine similarity with a minimum threshold δ, using k=1 in the main experiments.The system retrieves one Workflow entry, one Subtask segment per label, and one Function record per failing function, while deduplicating repeated Subtask segments.
- C Additional Implementation Details: Benchmark-specific evaluation covers 168 AppWorld test-normal tasks, 200 BFCL V3 multi-turn base tasks, and 129 ToolSandbox base tasks.AppWorld augments Function memory with argument and response schemas, whereas the other benchmarks provide minimal signature-derived information.
- D Memory Examples and Generation Prompts: The examples present Workflow, Subtask, and Function memories as structured, reusable knowledge distilled from successful teacher trajectories at different granularities.The case study illustrates planning a Venmo task, paginating and filtering payment requests, and extracting a float from a balance-response dictionary.
- D Memory Examples and Generation Prompts: Representative entries include a Workflow memory for playing a 2020 Spotify playlist song, a Subtask memory for retrieving liked songs, and a Function memory example.The examples identify involved apps, task or segment metadata, labels, and API-oriented execution details.
- D Memory Examples and Generation Prompts: The Workflow build prompt asks for retrieval-friendly memory stored only for successful tasks, based on the task instruction and the teacher’s Python trajectory and environment outputs.A representative Workflow memory specifies credential retrieval, note search, identity resolution, and subsequent transaction calls using placeholders.