Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

Julia Belikova; Rauf Parchiev; Evgeny Egorov; Grigorii Davydenko; Gleb Gusev; Andrey Savchenko; Maksim Makarenko

Source-linked AI summary

Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

Julia Belikova, Rauf Parchiev, Evgeny Egorov, Grigorii Davydenko, Gleb Gusev, Andrey Savchenko, Maksim Makarenko

arXiv:2606.23127v1Jun 22, 2026cs.AIcs.CLcs.SE

TL;DR

Reusable procedural skills are valuable for workplace agents but may fail when tasks, roles, or model backbones change. AFTER evaluates this transfer across 382 enterprise tasks and finds consistent gains alongside strong cross-model transfer from diverse multi-model traces, despite role-specific specialization.

Problem
Procedural memory’s ability to produce skills that transfer across tasks, roles, and model backbones remains poorly understood.
Method
AFTER benchmarks procedural skill transfer across 382 workplace tasks, six professional roles, 22 skills, and controlled task, role, and model shifts.
Results
73.1% cross-model test accuracy was achieved by skills evolved from diverse multi-model traces, while procedural memory improved accuracy across tasks but could specialize under cross-role transfer.
Takeaways & Limitations
Procedural memory systems should evaluate and develop skills for transfer across contexts while accounting for specialization to local workflows.
Takeaways & Limitations
AFTER focuses on technology-sector, procedural, tool-use workplace workflows and excludes open-ended creative or conversational tasks.

Abstract

from arXiv · show

Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones. The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Experiments show that procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points, while skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. We further find that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. These results provide practical guidance for building, evaluating, and deploying procedural memory systems in production agent platforms.

1 Introduction

The paper frames procedural memory as an evolving layer for reusable workplace-agent capabilities, whose value depends on transfer beyond the source context. It introduces AFTER and EVOLUTION to evaluate and control specialization, local improvement, and transfer across tasks, roles, and model backbones.

Motivation and problem: Procedural memory distills reusable instructions, procedures, and strategies from prior trajectories, but narrow experience may fail when models, tools, tasks, or workflows change.Its practical value depends on capturing knowledge that transfers across tasks, users, roles, and model backbones.
Specialization and generalization: Skills optimized from narrow experience can over-specialize to their source workflow, whereas diverse experience is expected to promote broader reuse.The paper presents specialization and generalization as complementary properties of procedural memory and illustrates their trade-off in Figure 1.
Benchmark and evaluation: AFTER contains 382 realistic workplace tasks across six professional roles and 22 procedural skills, with controlled splits for specificity and generality.The splits measure in-context gain, held-out-task transfer, cross-role transfer, and cross-model transfer.
Results: +2.8 points average full-pass accuracy comes from procedural skills on the static benchmark, while one refinement round adds a further +5.2 points across model scales.These results motivate evaluating both local improvement and transfer rather than treating skills as static artifacts.
Results: 73.1% test accuracy is achieved by skills evolved from diverse multi-model traces, exceeding the best single-model trace source by +13.7 points.The result supports diverse experience as a source of more reusable cross-model skills.

2 AFTER: A Benchmark for Skill Transfer

AFTER benchmarks procedural skill transfer in LLM agents using 382 realistic workplace tasks across six professional roles and 22 procedural skills. Its role-driven structure and controlled cross-task, cross-role, and cross-model splits distinguish transferability from task completion and retrieval quality.

Transfer evaluation: Controlled cross-task, cross-role, and cross-model splits evaluate whether procedural knowledge transfers across contexts.AFTER combines a role-driven task–skill structure with transfer-focused evaluation rather than measuring task completion alone.
Role structure: Role-specific instantiations make the same skill support different workflows, such as invoice extraction for DE, document ingestion for GenAI, or executive summarization for PM.Each role induces a characteristic task–skill distribution.
Task design: 318 single-skill and 64 multi-skill tasks test workflows requiring either one skill or two to three skills in an input–process–output structure.Skill annotations are fixed at task definition, separating skill quality from retrieval quality and enabling retrieval to be studied separately.
Skill representation: 22 reusable skills are represented as self-contained, versionable SKILL.md artifacts spanning five capability areas.The areas are document processing, data operations, ML and AI, infrastructure, and software engineering.

3 Methods

The methods formalize procedural-memory updates as operators that optimize value on target contexts, distinguishing specificity from transfer-based generality. EVOLUTION standardizes trace collection, skill versioning, updates, validation, promotion, rollback, and transfer measurement.

Formal update framework: A procedural-memory configuration Σ is updated to Σ′ by applying an operator U to a pool of execution traces D.U may instantiate reflection, distillation, or a learned memory-writing policy.
Formal update framework: The update rule seeks to maximize expected value on target contexts given an initial configuration, source and target distributions, and a trace budget.The admissible family U defines the available update mechanisms, while V(Σ; c) measures configuration value in context c.
Evaluation protocol: Specificity sets ptgt = psrc, whereas generality shifts the task, role, or model distribution between source and target contexts.These distribution settings provide controlled conditions for evaluating local improvement and transfer.
EVOLUTION harness: EVOLUTION standardizes trace collection, skill versioning, update execution, and transfer measurement through versioned SKILL.md artifacts with YAML metadata and markdown bodies.Each execution emits a trace linked to the active skill version, supporting reproducible updates and evaluations.
EVOLUTION harness: The harness supports full-skill updates through a COLLECT–DIAGNOSE–REVISE–PROMOTE cycle and applies s(v+1) = Uρ(s(v), Dsrc).The reflector ρ inspects traces, summarizes failure modes, and proposes a revised skill body; EVOLUTION fixes validation, promotion, rollback, and lineage tracking.

4 Experiments and Results

Experiments evaluate procedural memory through static skill content, single-round refinement, trace-based evolution, and transfer across models, roles, and inference efficiency. Results show consistent refinement gains, stronger cross-model transfer from diverse traces, role-specific specialization, and reduced token usage.

Experimental design: The evaluation proceeds through static skill testing, single-pass refinement, narrow versus diverse trace-based evolution, and transfer and efficiency analyses.Static skills are compared without orchestration, retrying, tool use, or evolution; later stages examine adaptation and generalization.
Static skills: +14.2 points on GenAI for Gemma 4 E4B versus +3.1 for GPT 5.4, while generated skills often outperform handcrafted skills.Static evaluation reports M2 as the primary metric and finds more consistent benefits for weaker models.
Single-round refinement: +3.7 to +6.7 aggregate points result from a single refinement round across model scales.The refinement converts handcrafted skills from Hpre to Hpost, with larger models benefiting more consistently and Infra and SWE showing the strongest gains.
Trace-based evolution: Large training gains do not necessarily improve held-out tasks, revealing a gap between specialization and transfer across narrow and diverse evolution.Table 3 compares five approaches under narrow n = 1 and diverse n = 5 evolution using shared experimental conditions.
Cross-model transfer: 73.1% cross-model test accuracy from diverse multi-model traces exceeds the 36.0–59.4% range from single-model sources.Weaker source models provide better transferable signal than stronger models, suggesting value from imperfect executions.
Cross-role transfer: +11.7 points for PM and +6.2 for DS in-role evolution contrast with -4.8 to -7.5 points under cross-role transfer.Role-specific purposes for the same skill, such as PDF extraction, lead procedural evolution to specialize by role.

5 Conclusion

AFTER evaluates procedural memory through transfer across tasks, roles, and model backbones. Across 382 workplace tasks, procedural skills improve accuracy, while diverse multi-model traces substantially improve cross-model generalization.

AFTER is a benchmark evaluating procedural memory through transfer across tasks, roles, and model backbones.
+2.8 points in full-pass accuracy came from procedural skills across 382 workplace tasks, with one skill-evolution round adding +5.2 points.
73.1% cross-model test accuracy was achieved by skills evolved from diverse multi-model traces, outperforming the best single-model trace source by at least +13.7 points.

Limitations

The study’s conclusions are limited by AFTER’s technology-focused task coverage, controlled trace budgets and automated correctness evaluation, and non-exhaustive model and framework selection.

Benchmark coverage: AFTER may underrepresent healthcare, legal, and scientific-research workflows because it targets technology-sector roles and workplace tasks partly drawn from the authors’ practice.Its 22 skills also exclude open-ended creative and conversational tasks, limiting conclusions to procedural, tool-use-oriented workflows.
Evaluation scope: The fixed trace budget enables controlled comparison but leaves the relationship between larger trace pools and transfer quality unresolved in real deployments.Deployments may accumulate substantially more traces than the experiments allow.
Evaluation scope: Automated pytest verification measures functional correctness but not code readability, robustness beyond the test suite, or user preference.These omitted qualities could matter in practical evaluation and deployment.
Model and framework selection: The evaluated LLMs and procedural-memory frameworks are representative but non-exhaustive because access constraints and benchmark finalization timing excluded several newer frontier systems.Recently released models and memory systems could not all be included.

Ethics Statement … D Benchmark Details

The paper uses public datasets and positions AFTER against prior agent and skill benchmarks by isolating procedural-memory transfer under controlled conditions. EVOLUTION makes skill evolution explicit and reproducible, while supporting multiple procedural-memory systems and context-specific adaptation strategies.

Ethics Statement: All datasets are public and were collected and preprocessed by their original authors.
A Related Work: Prior work spans reflective guidance, cross-task insights, replayed successes, tool documentation, optimized prompts, and discrete model-agnostic skill artifacts.
A Related Work: Existing benchmarks either score full agent pipelines or compare skill conditions, but do not jointly vary skill sources, role structure, annotations, and controlled transfer splits.
B Evolution Details: EVOLUTION records skill versions, traces, updates, evaluations, promotions, and rollbacks so procedural-memory evolution can be compared reproducibly under one protocol.
B Evolution Details: Each evolution iteration collects executions, diagnoses recurrent failures, revises the skill, and promotes candidates only when validation improves by at least margin δ.
B Evolution Details: Context-specific adapters prepend trace-distilled prefixes keyed by task, role, or model while leaving the shared skill body frozen; their empirical evaluation is deferred.
C External Agentic Frameworks with Procedural Memory: Four procedural-memory systems are evaluated through EVOLUTION, differing in how procedural knowledge is represented, selected, and updated.
C External Agentic Frameworks with Procedural Memory: EvoSkill, Memp, Hermes, and Memento-Skills respectively refine feedback-driven skills, distill editable procedures, generate structured YAML blueprints, and co-evolve routing with markdown skill writing.

D.1 Data Splits · D.2 Task Format · E Benchmark Construction

AFTER organizes each role–skill cell into controlled train, validation, and test folds, with additional role-level holdouts for cross-role transfer. Its tasks use a standardized enterprise-workflow format while intentionally leaving implementation procedures to procedural memory.

D.1 Data Splits: 50% train, 25% validation, and 25% test folds structure every role–skill cell for skill evolution, tuning, and final generalization evaluation.Train traces support skill evolution; validation supports hyperparameter tuning and overfitting prevention; test measures genuine generalization.
D.1 Data Splits: Cross-role transfer additionally holds out entire roles, such as training PDF skills on DE-PDF, DS-PDF, and PM-PDF before evaluating on unseen GenAI-PDF tasks.The held-out role is never seen during training in this evaluation setup.
D.2 Task Format: Each AFTER task includes metadata describing its name, role, required skills, difficulty, and data-source attribution.These fields are stored in task.toml.
D.2 Task Format: Each task provides realistic colleague-style instructions, authentic workplace-format inputs, and automated pytest-based output verification.Instructions are stored in instruction.md, inputs include Excel, PDF, CSV, and JSON files, and checks are defined in tests/test_outputs.py.
E Benchmark Construction: The benchmark’s standardized task format combines realistic requests, authentic files, and executable correctness checks to support enterprise-workflow evaluation.The format is defined through task metadata, instructions, inputs, and verification tests.
D.2 Task Format: Task instructions omit implementation details that procedural memory is expected to supply.An extraction request may name the transaction table and bank statement without specifying the library, multi-page handling, or output format.

E.1 Task Origins · E.2 Quality Assurance

AFTER combines adapted benchmark tasks, author-designed scenarios, and LLM-generated tasks under a unified workplace-task contract. Every task undergoes automated checks and independent two-reviewer inspection for clarity, skill fit, realism, verifier validity, determinism, and related quality criteria.

E.1 Task Origins: AFTER draws tasks from public benchmarks, author-written scenarios, and multi-stage LLM-based generation.This provides three complementary task-origin sources.
E.1 Task Origins: 56 tasks were adapted from 13 public benchmarks and source repositories while preserving each underlying problem and success criterion.Instructions were rewritten to satisfy the AFTER task contract, and verification was reimplemented in the unified pytest harness; unfaithful adaptations were dropped.
E.1 Task Origins: 38 tasks were written by the authors, including 18 longer multi-turn scenarios that probe procedural memory across several reasoning steps.The 18 scenarios comprise 3 per role.
E.1 Task Origins: The remaining 288 tasks were drafted by Claude Sonnet 4.6 and iteratively rewritten until they met the stated criteria.Each draft was scored before revision.
E.2 Quality Assurance: Every task was reviewed through a uniform protocol combining automated checks with two-reviewer inspection.The rubric covered clarity, skill fit, realism, dependency hygiene, verifier soundness, no oracle leakage, determinism, and self-containment.
E.2 Quality Assurance: Verifier soundness required rejection of empty, constant, and random adversarial baselines, while dependency hygiene required lightweight, easily installable dependencies.These requirements operationalize core automated quality checks.
E.2 Quality Assurance: Tasks also required no ground-truth values, hard-coded answers, or hidden hints, and self-containment in the task directory or through a fixed seed.These criteria prevent oracle leakage and external-material dependence.
E.2 Quality Assurance: Automated audits, calibration, adversarial gates, and independent dual review determined acceptance, with disagreements triggering rewriting and re-review.The automated checks covered metadata and paths, reference-solution calibration, adversarial outputs, and repeated-run consistency; human-review effort was approximately recorded in the source passage.

F Experimental Setup · G Token Usage for auto-agents

The experiments evaluate procedural memory across AFTER’s six roles and 22 skills using controlled baselines, refinement, evolution, transfer, and token-efficiency settings. Auto-agent measurements further assess how skill scenarios affect token usage, cost, and task success.

F Experimental Setup: AFTER evaluation covers all six professional roles and the complete 22-skill catalogue under the prescribed train, validation, and test splits.The test split contains unseen tasks for every role–skill combination.
F Experimental Setup: Experiments compare no-skill, handcrafted, and generated skill baselines with a single refinement pass and skill evolution.Evolved skills are evaluated for role transfer, model transfer, and token efficiency.
F Experimental Setup: Cross-role transfer focuses on pdf, xlsx, and validation, the three high-overlap skills shared by four roles each.These skills define the controlled cross-role transfer setting.
F Experimental Setup: The baseline panel spans open- and closed-weight models across family and size tiers, including GPT, GPT-oss, Qwen, Gemma, DeepSeek, and Nemotron.Skill rewriting additionally uses Claude Sonnet 4.6, GPT 5.5, and DeepSeek V4 Flash; cross-model transfer adds Qwen 3.5-27B and Llama 8B.
F Experimental Setup: The train split is used for evolution, while the test split contains unseen tasks and validation is treated as a separate split.The paper specifies train, test, and validation assignments for evaluation.
G Token Usage for auto-agents: Token-cost experiments run Claude Sonnet 4.6 and Hermes on three representative auto-agent tasks across no-skill, handcrafted, evolved, and self-evolved scenarios.Generated tokens, total tokens, and dollar cost are averaged over four runs.
G Token Usage for auto-agents: 521k→195k total tokens for Claude on Kafka Lag Anomaly Detection illustrates that evolved skills can reduce usage relative to the handcrafted baseline.The reduction lowers cost without sacrificing task success.

H Reflector Ablation … L Example Skills

The paper evaluates skill evolution across reasoners while holding the solver fixed, and contrasts narrow handcrafted skills with broad LLM-generated procedural references. The accompanying benchmark tables define train-split, full-result, and single-refinement evaluations across roles and skill conditions.

H Reflector Ablation: Diverse training beats narrow training for every reasoner when evolving pptx and xlsx skills with GPT-oss-120B held fixed.The reasoners are Claude, Hermes, Codex, and a script without an agent; each tunes a handcrafted baseline on n = 1 to n = 5 tasks and is tested on 3 held-out tasks.
H Reflector Ablation: The ablation attributes the performance gain to evolution itself rather than to any particular agent.
I Train Split Results: Train-split results report average task M1 per role across 185 tasks under no-skill, handcrafted, and LLM-generated skill conditions.Per-task M1 is the mean over attempts of tests_passed/task_total; role and aggregate values are unweighted task means.
J Full Results: Full results report per-role M1 and M2 under no-skill, handcrafted, and LLM-generated conditions, including aggregate full-pass rates and best skill deltas.The M2 table defines per-task pass rate as the mean over attempts of tests_passed/task_total and aggregates role values as unweighted task means.
K Single Refinement Results: Single-refinement results compare canonical handcrafted skills with single-pass Codex-refined skills using average M2 across 111 task IDs present in both runs.Hpre denotes canonical handcrafted skills, while Hpost denotes Codex-refined skills; Hpost cells are colored by their change.
L Example Skills: Every paper-target skill ships with a handcrafted body authored for a concrete user-facing scenario and an LLM-generated body drafted as a broad procedural reference.The paper illustrates this contrast with the sql skill.
L Example Skills: The handcrafted sql skill targets one narrow task: parsing an unknown SQLite database into structured JSON through schema exploration, relationship analysis, extraction, and output shaping.The example includes table and schema inspection, key discovery, foreign-key analysis, row-to-JSON extraction, and debugging guidance.
L Example Skills: The LLM-generated sql skill is a seven-section cheatsheet covering SQL techniques, optimization, access libraries, connection patterns, worked examples, and efficient access patterns.Its reference includes joins, window functions, CTEs, EXPLAIN ANALYZE, index design, and DuckDB, sqlite3, SQLAlchemy, and psycopg2 access layers.