PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

Ziyan Liu, Zhezheng Hao, Yeqiu Chen, Hong Wang, Jingren Hou, Ruiyi Ding, Yongkang Yang, Wence Ji, Wei Xia, Feng Liu

arXiv:2605.30159v1cs.AI

TL;DR

Existing memory policies rely on sparse outcome rewards that do not localize degradation in recursive summaries. MMPO uses Belief Entropy for dense intermediate supervision, and experiments show consistent improvements over outcome-based memory RL baselines across long-horizon tasks.

  • Problem

    Outcome-based training provides sparse supervision and lacks a principled criterion for localizing and optimizing intermediate memory degradation during recursive summarization.

  • Method

    MMPO uses Belief Entropy, a metacognitive proxy for summary-induced belief uncertainty, to provide dense supervision for intermediate memory states.

  • Results

    MMPO consistently improves over outcome-based memory reinforcement-learning baselines across long-horizon agent tasks and improves accuracy over RL-MemAgent across long-context settings.

  • Takeaways & Limitations

    Belief Entropy provides useful intermediate supervision for recursive memory summarization and supports clearer summary-induced beliefs across diverse long-horizon tasks.

  • Takeaways & Limitations

    Belief Entropy is only a proxy whose effectiveness depends on task-relevant anchor questions and whether response uncertainty reflects information preserved in memory.

Abstract

from arXiv · show

Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.

1 Introduction

Long-horizon LLM agents suffer from semantic noise and belief deviation as recursive summaries accumulate, while outcome-based training provides insufficient supervision for intermediate memory quality. MMPO addresses this by penalizing ambiguous summaries and using Belief Entropy to preserve clearer summary-induced beliefs.

  • Problem: Recursive summarization accumulates semantic noise that can trigger cascading hallucinations, memory explosion, and performance decay over extended interactions.The problem is attributed to the absence of a principled criterion for optimizing intermediate summaries.
  • Problem: Outcome-based RLVR summary policies provide sparse credit assignment and no explicit supervision to suppress noise accumulation during recursive summarization.This leaves agents prone to retaining noisy or irrelevant information in memory.
  • MMPO: MMPO improves long-horizon reasoning by penalizing ambiguous intermediate summaries and preserving belief clarity.Its anchor-question-based Belief Entropy provides dense, memory-specific supervision for epistemic uncertainty.
  • Motivation: Reliable long-horizon memory requires fine-grained supervision of the clarity of beliefs induced by intermediate summaries, not only final task outcomes.The POMDP formulation treats hidden task states as requiring action according to an internal belief state derived from interaction history.

2 Belief Entropy

Belief Entropy measures uncertainty about the latent task state from a compressed memory, operationalizing belief preservation for long-horizon agents. Lower entropy tracks clearer summaries and better task outcomes, enabling inference-time trajectory selection.

  • Summary-Induced Belief: Summary-based agents compress interaction history into bounded memory, inducing their belief about the hidden task state from that compressed representation.This replaces full-history conditioning, which becomes impractical as long-horizon interactions grow and belief deviation accumulates.
  • Belief-Preservation Objective: Memory optimization should minimize conditional entropy H(s_t | m_t), making summaries informative representations that preserve a reliable belief about the latent task state.This objective is equivalent to maximizing mutual information between the summary and latent state.
  • Belief Entropy: Belief Entropy estimates otherwise unobservable task-state uncertainty by measuring the model’s response uncertainty to a memory-conditioned anchor question.Clear memories should produce concentrated responses, whereas ambiguous or incomplete memories should yield higher uncertainty.
  • Anchor Question: The dual-probe anchor asks for current task progress and remaining information needs, targeting task-state uncertainty rather than generic model confidence.Its progress and information-gap components expose the current state estimate and residual uncertainty.
  • Empirical Validation: Pearson r = −0.684 links total entropy reduction to final task accuracy, while selecting the lowest-entropy trajectory among N = 5 candidates improves accuracy over Vanilla+Memory.Successful trajectories generally decrease in HBE, whereas failed trajectories stagnate or increase, and HBE supports training-independent inference-time selection.

3 Metacognitive Memory Policy Optimization (MMPO)

MMPO optimizes recursive memory policies with dense, intermediate supervision from Belief Entropy while retaining final-task rewards. It assigns group-relative credit across sub-trajectories and future turns, then updates summary tokens with a clipped PPO objective.

  • Reward design: MMPO combines intermediate belief quality with final task outcomes to provide dense process supervision for long-horizon memory-policy optimization.Belief Entropy supplies the intermediate signal, while the terminal reward anchors each sub-trajectory to task success.
  • Credit assignment: MMPO samples trajectory groups, computes per-turn Belief Entropy rewards, standardizes sub-trajectory rewards with GRPO, and aggregates them into future-aware turn-level advantages.The training pipeline decomposes trajectories into sub-trajectories and computes dense rewards at every turn.
  • Reward design: Lower Belief Entropy produces a larger bounded intrinsic reward, while higher entropy produces a smaller reward through sigmoid normalization.The coefficient α controls the intrinsic belief reward’s relative weight against the outcome reward.
  • Credit assignment: A summary’s turn-level advantage averages the advantages of all later sub-trajectories that contain that turn, reflecting its influence on subsequent states.This aggregation reinforces summaries that lead to consistently clearer and more successful future states than group alternatives.
  • Policy optimization: MMPO distributes each turn-level advantage across summary tokens and optimizes them with a clipped PPO surrogate penalized against a reference model.The objective targets summaries that minimize belief deviation and maximize task-success probability across long horizons.

4 Experiments

Experiments evaluate MMPO against MemAgent and MEM1 across long-context, multi-objective QA, and WebShop settings while preserving the corresponding memory workflows. MMPO improves long-context accuracy, complements outcome-based training, and links decreasing Belief Entropy with successful reasoning.

  • Comparison with MemAgent: MMPO improves average RULER-HotpotQA accuracy over RL-MemAgent for both Qwen2.5-7B and Qwen2.5-14B backbones.The comparison uses the same recursive memory workflow as MemAgent under controllable long-context scaling.
  • Comparison with MemAgent: +3.14% average accuracy is the MMPO gain over RL-MemAgent for Qwen2.5-7B from 224K to 3.5M context length.The corresponding Qwen2.5-14B gain is reported as +3.12% over the same context-length range.
  • Comparison with MEM1: MMPO improves over MEM1-QA across evaluated objective horizons, with larger gains on harder 8-objective and 16-objective QA settings.These settings require maintaining multiple unresolved information needs over longer trajectories.
  • Comparison with MEM1: MMPO also improves over MEM1-WebShop, indicating that its supervision extends beyond retrieval-focused tasks.The comparison tests whether Belief Entropy supervision complements MEM1’s outcome-based memory training.
  • Belief Entropy Dynamics: Successful trajectories generally show decreasing HBE as evidence accumulates, whereas failed trajectories tend to stagnate or increase.Across 500 test episodes, MMPO strengthens the correlation between total entropy reduction ∆HBE and task accuracy compared with MemAgent.

5 Related Work

Related work spans memory-augmented agents, reinforcement learning for LLMs, and belief-state modeling under partial observability. These lines of research motivate compact memory management, intermediate-step credit assignment, and uncertainty-aware reasoning for long-horizon agents.

  • Memory-Augmented LLM Agents: Memory-augmented agents address long interaction histories through truncation, retrieval-augmented context, trajectory compression, and learnable memory-management layers.Representative systems include MemGPT’s operating-system-inspired memory hierarchy, alongside Mem0 and MemOS.
  • Reinforcement Learning for LLMs: Outcome-based reinforcement learning methods such as RLHF are complemented by process supervision that assigns rewards to intermediate reasoning states.PRM, PRIME, and Miracle target multi-step credit assignment, while GRPO uses group-relative normalization without a value model.
  • Belief States and Uncertainty Estimation: Belief-state research studies decision-making under partial observability, history compression, and information-theoretic sufficient statistics.Recent work identifies belief deviation as a key failure mode of long-horizon LLM agents.

6 Conclusion · A MMPO Algorithm · B Summary-Induced Belief: Architectural Justification

MMPO optimizes long-horizon agent memory using Belief Entropy to supervise intermediate memory reliability alongside final task success. The appendices specify its trajectory-level PPO procedure and justify summary-induced belief as the action policy’s memory-conditioned state estimate, which is information-limited by summarization.

  • 6 Conclusion: MMPO uses Belief Entropy to estimate uncertainty in summary-induced beliefs and provide dense supervision for intermediate memory states.This extends memory optimization beyond final task success to the reliability of evolving memory.
  • 6 Conclusion: MMPO consistently improves over outcome-based memory reinforcement-learning baselines on long-horizon agent tasks.
  • A MMPO Algorithm: The algorithm samples N complete trajectories per task, generates memory summaries at each turn, and computes token-level predictive entropy for those summaries.It then obtains terminal outcome rewards and evaluates sub-trajectory rewards across depths.
  • A MMPO Algorithm: MMPO normalizes group-relative advantages, aggregates turn-level advantages, and updates the policy with a clipped PPO objective.The procedure uses policy πθ, reference policy πref, group size N, maximum turns T, and coefficients α, β, ϵ.
  • B Summary-Induced Belief: Architectural Justification: The summary-based architecture conditions downstream reasoning and action selection on textual memory mt rather than the full interaction history ht.This defines the belief used by the agent without claiming that mt is a sufficient statistic of ht.
  • B Summary-Induced Belief: Architectural Justification: I(st; mt) ≤ I(st; ht), so summarization cannot increase information about the latent task state.State information not preserved in mt is unavailable to the action policy because downstream reasoning no longer conditions on the original history.
  • B Summary-Induced Belief: Architectural Justification: Summary-induced belief mixes full-history beliefs from histories that could produce the same memory mt.Ambiguous summaries can combine histories requiring different decisions, causing belief deviation through semantic noise, omitted evidence, or conflated entities.

C Information-Theoretic Justification for Belief Entropy · D Details of Compared Memory-Agent Frameworks

Belief Entropy approximates memory-induced uncertainty about latent task state through the entropy of an anchor-question response, providing dense intermediate supervision for memory clarity alongside final-task rewards. The compared MemAgent and MEM1 frameworks maintain compact recurrent or internal memories to support reasoning over inputs and interactions exceeding native context limits.

  • C Information-Theoretic Justification for Belief Entropy: Belief Entropy targets the ideal objective of minimizing conditional uncertainty H(s_t | m_t) about the latent task state given current memory.Because s_t is unobservable in open-ended LLM-agent settings, the method probes this uncertainty through a state-relevant anchor question and the model’s response distribution.
  • C Information-Theoretic Justification for Belief Entropy: The anchor response probes preserved task-state information through current progress and unresolved information gaps rather than serving as an arbitrary model output.The response is conditioned on the current memory m_t and anchor question q.
  • C Information-Theoretic Justification for Belief Entropy: H_BE(m_t) = H(y | m_t, q), with lower response entropy indicating task-relevant information preservation and higher entropy indicating omitted evidence or semantic noise.A concentrated response distribution suggests that the memory supports answering the anchor question with less uncertainty.
  • C Information-Theoretic Justification for Belief Entropy: The chain-rule decomposition separates state-conditioned response uncertainty from residual task-state uncertainty exposed through the anchor response.The decomposition is H(y | m_t, q) = H(y | m_t, q, s_t) + I(y; s_t | m_t, q).
  • C Information-Theoretic Justification for Belief Entropy: Belief Entropy is an anchor-probed proxy rather than the full H(s_t | m_t), but lower response uncertainty indicates more reliable resolution of task-state information exposed by the question.The residual term satisfies I(y; s_t | m_t, q) ≤ H(s_t | m_t, q) when comparing memories at the same turn.
  • C Information-Theoretic Justification for Belief Entropy: MMPO combines dense Belief Entropy credit for memory clarity with verifiable outcome rewards anchoring optimization to final task success.This avoids optimizing solely for response confidence rather than useful memory content.
  • D Details of Compared Memory-Agent Frameworks: MemAgent recurrently updates memory from the task query, previous memory, and current context segment, then generates the final answer from accumulated memory.This fixed-window process supports inputs longer than the model’s native context length.
  • D Details of Compared Memory-Agent Frameworks: MEM1 maintains compact internal memory across steps for task-directed reasoning, querying, or environment interaction, and is evaluated on multi-objective QA and WebShop-style interactive tasks.The framework preserves multiple information needs or actions across long-horizon interactions.

E More Results … E.3 Computational Overhead

The additional results show that progress-and-gap anchoring best supports Belief Entropy, which aligns more closely with task accuracy than alternative proxy signals. MMPO incurs modest training overhead, while inference can omit Belief Entropy and retains unchanged peak GPU memory.

  • E.1 Full Anchor Question Robustness Study: The direct-answer anchor underperforms Outcome Only on average, whereas gap-only and progress+gap probes better target intermediate memory quality.Directly rewarding answer confidence can encourage premature certainty when memory is incomplete.
  • E.1 Full Anchor Question Robustness Study: The progress+gap probe is the strongest anchor design because it explicitly tracks task progress and remaining information needs.Its prompt asks what the current task progress is and what information is still needed.
  • E.2 Comparison with Alternative Proxy Signals: Belief Entropy achieves the strongest correlation with task accuracy and the best downstream performance among evaluated low-cost proxy signals.The comparison uses RULER-HotpotQA with Qwen2.5-7B at 56K context length; |r| denotes absolute Pearson correlation with task accuracy.
  • E.2 Comparison with Alternative Proxy Signals: Anchor-question Belief Entropy better aligns with intermediate memory quality than generic confidence, memory length, or random-question entropy.Selected-token NLL lacks direct task-state uncertainty probing, while direct-answer entropy may encourage premature confidence before sufficient evidence is collected.
  • E.3 Computational Overhead: 12% training overhead results from one additional forward pass per turn for Belief Entropy computation on Qwen2.5-7B.The extra pass generates an anchor-question response.
  • E.3 Computational Overhead: 5% inference overhead applies only when Belief Entropy is used as an optional confidence signal.Standard decoding does not require Belief Entropy, so the additional inference cost can be removed.
  • E.3 Computational Overhead: Peak GPU memory remains unchanged because the anchor-question pass reuses the same model.This memory result holds alongside the reported optional confidence-signal inference overhead.

F Implementation Details

MMPO follows MemAgent’s recurrent memory workflow, updating summaries from the query, prior memory, and document chunks. Training uses group-relative policy optimization with specified optimization, rollout, and Belief Entropy settings.

  • Memory Generation: At each turn, MMPO generates an updated memory from the task query, previous memory, and current document chunk using MemAgent’s recurrent workflow.The context allocates 1,024 tokens to the query, 5,000 to the document chunk, 1,024 to memory, and 1,024 to memory output.
  • Training Setup: MMPO training uses AdamW with a 1×10−6 learning rate, constant schedule with linear warm-up, 1×10−3 KL coefficient, disabled entropy loss, and G = 16.The Belief Entropy reward weight is α = 0.5.
  • Training Setup: Each rollout batch contains 128 trajectories from 8 prompts, with 16 sampled trajectories per prompt.This corresponds to the group size G = 16 used by group-relative policy optimization.
  • Prompt Templates: The memory-update prompt instructs the model to retain relevant prior details while incorporating useful information from the current article section.The prompt provides a problem, article section, and previous memory for each update.
  • Prompt Templates: For QA tasks, Belief Entropy uses an anchor question asking for current task progress and the information still needed.The anchor is conditioned on the problem and current memory.

G Limitations · H Impact Statement

The paper presents Belief Entropy as a practical proxy whose validity depends on task-relevant anchor questions and response uncertainty, while MMPO improves supervision without guaranteeing factual correctness or safe autonomy. It highlights potential benefits for sustained-context applications alongside deployment requirements for privacy, retention, and transparency.

  • G Limitations: Belief Entropy proxies summary-induced belief uncertainty rather than directly measuring latent task-state uncertainty.Its effectiveness depends on whether the anchor question captures task-relevant uncertainty and whether response uncertainty reflects information preserved in memory.
  • G Limitations: The method mitigates proxy limitations by combining a task-state anchor question with verifiable outcome rewards.
  • H Impact Statement: MMPO aims to improve long-horizon agent reliability through denser supervision for intermediate memory states.
  • H Impact Statement: Improved memory optimization may benefit long-document reasoning, multi-step question answering, interactive search, and task-oriented assistants.
  • H Impact Statement: MMPO does not guarantee factual correctness or safe autonomous behavior.Inaccurate memories, overconfident summaries, or poorly designed anchor questions may still produce incorrect decisions.
  • H Impact Statement: Incorrect decisions remain a concern especially during long interactions or in high-stakes settings.
  • H Impact Statement: Memory-based agents may process sensitive user information, requiring privacy controls, retention policies, and user-facing transparency in practical deployments.
Loading 2605.30159v1…