Learning to Accumulate Knowledge with Mutual Information

1 INTRODUCTION

• Training a curator for a fixed executor requires accounting for both individual-entry utility and what entries contribute as the knowledge bank accumulates.Task-success-only GRPO can favor general guidance with substantial informational redundancy across entries.Source: §1 · p. 2
• The curator extracts reusable knowledge from agent trajectories, while the executor incorporates retrieved entries without updating its parameters.Prior work reports that high-quality, task-specific knowledge can improve agent performance.Source: §1 · pp. 1–2
• Effective accumulation seeks limited redundant overlap among entries alongside substantial incremental utility as the bank grows.General guidance may help individual tasks while adding limited information across entries.Source: §1 · p. 2, Figure 1 · p. 2
• Knowledge Weaver combines token-level MI-inspired feedback with marginal success rewards to preserve source-specific information and encourage gains beyond retrieved knowledge.Standalone success rewards also encourage entries to be useful individually.Source: §1 · p. 2
• At k = 10, success rates reach 54.0% on ALFWorld and 42.0% on WebShop, exceeding task-success-only GRPO by 16.9 and 18.7 percentage points, respectively.The evaluation supplies retrieved entries as guidance to a frozen executor.Source: §1 · pp. 2–3

2 RELATED WORK

• Knowledge construction includes organized memories, distilled lessons, conditional guidelines, workflows, and executable skills for later retrieval and reuse.Examples include reflective lessons and strategies (Majumder et al., 2023) and workflows (Wang et al., 2024).Source: §2 · p. 3
• RL approaches learn skill use or construction through skill internalization, task-specific utilization, joint library evolution, or curation credit from later execution outcomes.SkillRL couples skill distillation and library evolution with executor training (Xia et al., 2026), while SkillRise assigns curation credit through subsequent execution outcomes (Yao et al., 2026).Source: §2 · p. 3
• Prior MI applications encourage predictive representations, interpretable factors, distinguishable behaviors, or context-dependent dialogue and reasoning.Knowledge Weaver applies token-level MI-inspired feedback to encourage experience-specific content in curated entries.Source: §2 · p. 3

3.1 PRELIMINARIES

• A curator samples trajectory contexts to generate entries for a bank, while a frozen executor solves tasks using entries retrieved once from the query and held fixed during execution.The evaluation bank is generated separately after training.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 EFFECTIVE KNOWLEDGE ACCUMULATION

• The coverage model represents each entry's utility over an idealized task coordinate and predicts lower bank coverage when profiles overlap more at fixed per-entry utility.It assumes ideal selection without retrieval errors or interactions between entries; an entry adds coverage only where its utility exceeds the existing envelope.Source: §3.2 · p. 4
• With standalone success alone, shared outcome credit gives no bank-relative preference to an equally successful complementary entry over one for an already-supported task type.A covered routine can receive positive GRPO advantage, although an uncovered task type would extend the bank's capabilities.Source: §3.2 · pp. 4–5

3.3 MARGINAL SUCCESS FEEDBACK

• Marginal success-rate feedback measures task-success change when a candidate is added to entries retrieved from a fixed bank snapshot.Standalone success is evaluated separately; the candidate is appended without replacing retrieved entries, and execution gains can be negative under interference.Source: §3.3 · p. 5

3.4 MI-INSPIRED INFORMATION FEEDBACK

• Token-level MI-inspired feedback compares a generated token's likelihood under its source context with its mean likelihood under alternative contexts, crediting stronger contrasts in knowledge-body tokens.Generic tokens with similar probabilities across contexts receive little credit, while tokens with larger contrasts receive more.Source: §3.4 · p. 5
• The combined feedback adds MI token credit and marginal success to standalone success, with a formatting reward, and updates only the curator using GRPO.Invalid candidates receive only a formatting penalty and no MI credit.Source: §3.4 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• The evaluation covers ALFWorld's valid_unseen split across six task types and 100 sampled WebShop test tasks, retrieving top-k entries once before the first action.Baselines include prompt-based, established or human-written, RL-trained, and LLM-curated banks.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• At k = 10, Knowledge Weaver reaches 54.0% ALFWorld SR and 42.0% WebShop SR, beating GRPO by 16.9 and 18.7 percentage points, respectively.WebShop mean environment score is 66.4 versus GRPO's 49.0; Knowledge Weaver also exceeds SkillRL's ALFWorld SR by 2.8 percentage points.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 KNOWLEDGE BANK QUALITY ANALYSIS

• At k = 10, retrieved context averages approximately 660 tokens on ALFWorld and 594 on WebShop, yet Knowledge Weaver achieves higher SR than human-written banks on both benchmarks.These context lengths are less than half the corresponding human-written context lengths.Source: §4.3 · p. 7, Figure 3 · p. 8
• At k = 10, Knowledge Weaver's mean pairwise retrieval overlap is 12.92% versus GRPO's 13.58% on ALFWorld and 15.35% versus 24.48% on WebShop.On WebShop, Qwen3.5-4B, GPT-5, Claude, and GRPO remain below their own no-knowledge controls at every positive retrieval count; the curves cannot distinguish interference from retrieval mismatch or other content effects.Source: §4.3 · pp. 7–8

4.4 ABLATION STUDIES

• Adding token-level MI to SR-only training raises ALFWorld SR from 37.1% to 46.0% and WebShop SR from 23.3% to 40.7%.Adding MSR then reaches 54.0% ALFWorld SR (+8.0 percentage points) and 42.0% WebShop SR (+1.3 points).Source: Table 2 · p. 8, §4.4 · p. 8
• Across eight frozen executors, Knowledge Weaver improves ALFWorld SR and both WebShop metrics over no-knowledge execution for all eight.Versus GRPO, it has higher ALFWorld SR and WebShop score for seven of eight executors on each metric, and higher WebShop SR for six, with one tie.Source: Table 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

• Knowledge Weaver’s banks outperform evaluated baselines with a frozen executor, with gains across retrieval sizes and transfer to other executors that varies by model.Ablation indicates that MI adds value beyond task success alone.Source: §5 · p. 9
• Future work includes jointly training curator and executor, developing banks that generalize across models, and testing longer-horizon tasks with continual bank updates.The experiments keep the executor frozen; transfer gains vary by model.Source: §5 · p. 9
• The authors used generative AI to assist with software implementation and analysis, but not to formulate the core ideas, hypotheses, or methodological design.They state that AI-assisted code, analyses, text, and visual elements were reviewed, with analyses independently checked against original outputs.Source: AI USE STATEMENT · pp. 9–10
• The reproducibility section directs readers to training signals, evaluation protocols, prompts, hyperparameters, pseudocode, and baseline implementations.Human-written baseline entries came from volunteers who participated voluntarily and gave informed consent.Source: REPRODUCIBILITY STATEMENT · p. 10, ETHICS STATEMENT · p. 10

A PROOFS

• The proof appendix specializes Gaussian utility profiles with fixed centers, common height, and common scale to analyze envelope coverage.It denotes adjacent center gaps by ∆i and the standard normal density and cumulative distribution function by ϕ and Φ.Source: A PROOFS · p. 14

A.1 PROOF OF PROPOSITION 1

• Changing h rescales every profile by the same positive factor, so that factor can be taken outside the envelope integral.Source: §A.1 · p. 14
• The coverage coefficient is positive and finite because the envelope is bounded by individual profiles and their sum, whose unit-height integrals are finite.Source: §A.1 · p. 14
• Coverage is linear and strictly increasing in common height h for fixed width.The proof factors the common positive height out of the pointwise maximum and integral, then establishes the resulting coefficient is positive and finite.Source: §A.1 · p. 14

A.2 PROOF OF PROPOSITION 2

• At fixed per-entry mass, coverage strictly decreases as Gaussian scale σ increases, while coverage efficiency falls from a limit of 1 to 1/K; added-entry marginal coverage increases with center separation.For nonnegative integrable profiles, marginal coverage is the integral of the positive part of the candidate profile minus the existing envelope, without requiring Gaussian profiles.Source: §A.2 · pp. 14–15

B CASE STUDIES OF KNOWLEDGE BANK CONTENT

• The case study inspects complete entries from step-40 GRPO and Knowledge Weaver banks, each containing 50 entries; the examples do not directly assess bank-wide redundancy.Source: B CASE STUDIES OF KNOWLEDGE BANK CONTENT · p. 16, Knowledge Weaver | Step 40 · p. 16

GRPO | Step 40

• The selected GRPO entries repeat a possession-before-placement routine and general color-checking guidance; 48 of 50 entries begin with “Before”.Source: Knowledge Weaver | Step 40 · pp. 16–17, GRPO | Step 40 · pp. 16–17

Knowledge Weaver | Step 40

• The selected Knowledge Weaver entries cover distinct operations, including appliance use, applying product attributes, specifying search constraints, and continuing to another page.Source: Knowledge Weaver | Step 40 · pp. 16–17

C CURATOR AND EXECUTOR PROMPTS

• The curator receives four same-task-type trajectories and produces one skill entry of at most 100 words; fixed-bank evaluation keeps retrieved entries fixed during an episode.The prompts specify runtime trajectory, observation, and guidance fields, with task-specific input and action-list limits.Source: C CURATOR AND EXECUTOR PROMPTS · pp. 17–18

D BASELINE REPRODUCTION

• Skill1, SkillRise, and Mem0 banks were reproduced with Qwen3.5-4B and evaluated using a frozen Qwen3.5-4B executor, top-10 retrieval, and a shared protocol.Skill1 and SkillRise results are means and standard deviations across three runs.Source: D BASELINE REPRODUCTION · p. 20

D.1 SKILL1

• The Skill1 reproduction retained its query generation, reranking, library updates, and distillation objective, while excluding execution responses from the policy loss.Complete trajectories still supplied outcome rewards, and step-40 checkpoints generated the evaluation banks.Source: §D.1 · p. 21

D.2 SKILLRISE

• SkillRise reproduction retained cross-task training and decoupled credit assignment, updating skills between three same-family tasks and resetting between trials.Curation responses received feedback from subsequent tasks; solving responses received their own task outcomes.Source: §D.2 · p. 21

D.3 MEM0

• Mem0 v2.1.0 semantic and procedural memories were exported from matched trajectory bundles into the shared bank format and evaluated across three seeds.Results averaged the two bank types for each seed and report means and standard deviations.Source: §D.3 · p. 21

D.4 HUMAN KNOWLEDGE CONSTRUCTION

• Volunteers wrote human-baseline entries from sampled trajectories, which were assembled into banks and evaluated under the shared protocol.Participation was voluntary and informed consent was obtained.Source: §D.4 · p. 21

E.1 HYPERPARAMETER DETAILS

• Training updates all curator parameters while keeping the executor and reference model frozen; the trained curator then generates evaluation banks.Table 4 summarizes the Qwen3.5-4B curator configuration.Source: §E.1 · p. 21

• The full training split is eligible for reward sampling, but each group samples eight tasks rather than evaluating the entire split at every update.Source: §E.2 · p. 21
• For ALFWorld, training-side sources belong to train, while final evaluation uses valid_unseen.Source: §E.2 · p. 21
• For WebShop, official human training and test tasks have disjoint task IDs, and synthetic bank-generation tasks use a separate ID namespace.Source: §E.2 · p. 21

• Only curator parameters are updated; the executor and reference model remain frozen, and the separately generated evaluation banks contain 50 entries with k = 10 in the main comparison.Source: §E.3 · p. 21, Table 4 · p. 22
• Each training batch contains 32 trajectory contexts and eight candidates per context, yielding 256 candidate entries before optimization.Source: Table 4 · p. 22, §E.3 · p. 22
• Execution feedback uses eight reward-task rollouts per candidate and condition, with matched tasks across standalone, bank-only, and bank-plus-candidate conditions; standalone and marginal success each receive weight 0.5.Source: §E.3 · p. 23
• The training bank starts empty and stays fixed during batch scoring; admission requires positive marginal success gain, with at most one WebShop entry per batch and scheduled ALFWorld quotas.Source: §E.3 · p. 23

• The evaluation varies retrieval count k from 0 to 15 while holding each bank, executor, and benchmark task set fixed.Source: §E.4 · p. 23
• 54.0% ALFWorld and 42.0% WebShop success at k = 10; at k = 15, the corresponding rates are 54.2% and 43.7%.Source: Figure 4 · p. 23, §E.4 · p. 23
• Each curve uses three fixed runs across retrieval counts and retains its own no-knowledge control at k = 0.Source: Figure 4 · p. 23, §E.4 · p. 23

F TRAINING ALGORITHM

• Token-level MI feedback rescores each candidate across source contexts with its generated tokens and prefix fixed, then adds local credit after return computation.Source: F TRAINING ALGORITHM · pp. 23–24, §F.1 · p. 24

F.1 OPTIMIZATION DETAILS

• The curator is updated with a clipped GRPO surrogate using fixed advantages and a detached proximal probability; the executor and reference policy remain frozen.Source: F TRAINING ALGORITHM · p. 23, §F.1 · p. 25

F TRAINING ALGORITHM

• Candidates from each trajectory context form a GRPO group; execution scoring combines standalone utility, marginal success, and formatting reward before within-group normalization.Source: F TRAINING ALGORITHM · p. 24
• Bank admission selects valid candidates absent from the snapshot with positive marginal success, ranks them by gain, and applies the task-type quota schedule.Source: F TRAINING ALGORITHM · p. 24

F.1 OPTIMIZATION DETAILS

• Execution rewards are normalized across candidates from the same source context and placed at the final generated token; sampled KL penalties accumulate backward across token positions.Source: §F.1 · p. 24
• The combined execution and MI advantages receive reference regularization and batch normalization over unmasked generated tokens before the update.Source: §F.1 · pp. 24–25