PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks

Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, Jitao Sang

arXiv:2510.12635v3cs.AI

TL;DR

Long-horizon agents need to curate working memory because unmanaged context causes attention dilution, while existing methods decouple memory management from reasoning. MemAct makes memory management learnable policy actions and uses DCPO to train with dynamic context updates. MemAct-RL-14B matches larger models while using substantially less context, and the paper identifies lossy compression and sparse-reward credit assignment as important boundaries.

  • Problem

    Unmanaged long-horizon context becomes saturated with irrelevant information, while externally controlled memory management is decoupled from the reasoning policy.

  • Method

    MemAct treats working-memory editing as actions in a unified policy, with DCPO segmenting trajectories to support end-to-end reinforcement-learning optimization.

  • Results

    MemAct-RL-14B matches models over 16× larger while reducing average context length by 51% and establishing a superior accuracy-efficiency Pareto frontier.

  • Takeaways & Limitations

    Learned policies discover capacity-aware memory strategies that generalize across task complexity and domains.

  • Takeaways & Limitations

    Sparse final-output rewards make it difficult to assign credit to individual memory actions, and lossy compression prevents recovery of summarized details.

Abstract

from arXiv · show

Long-context Large Language Models, despite their expanded capacity, require careful working memory management to mitigate attention dilution during long-horizon tasks. Yet existing approaches rely on external mechanisms that lack awareness of the agent's reasoning state, leading to suboptimal decisions. We propose Memory-as-Action (MemAct), a framework that treats working memory management as learnable policy actions. By formulating context management as in-place editing operations (deletion, insertion), MemAct enables joint optimization of information retention and task performance through end-to-end reinforcement learning. To address the computational challenges of dynamic context updates, we introduce Dynamic Context Policy Optimization, which restores training efficiency without compromising reasoning integrity. Experiments show that MemAct-RL-14B matches the accuracy of models $16\times$ larger while reducing average context length by 51\%, with learned strategies that adapt to model capabilities and generalize across task complexities.

1 Introduction

MemAct treats context curation as a learnable part of the agent’s policy, addressing attention dilution and the limits of externally controlled memory management. The framework combines memory-editing actions with end-to-end reinforcement learning and introduces DCPO to make dynamic-context training feasible.

  • Motivation: Long-horizon agents suffer attention dilution when unmanaged context becomes saturated with irrelevant information, burying critical signals.This motivates active context curation rather than relying only on larger context windows.
  • Motivation: Existing context-management designs decouple memory management from the reasoning policy, preventing joint optimization of information retention and task performance.These approaches commonly rely on heuristic rules or external controllers.
  • Memory-as-Action: MemAct makes context curation intrinsic by placing task actions and memory actions in a unified policy space.The agent can retain, compress, discard, or synthesize history through explicit function-call actions.
  • Training Method: DCPO restores training feasibility by segmenting fractured trajectories so dynamic context updates can be optimized with standard reinforcement-learning infrastructure.Memory edits otherwise violate causal language models’ assumption of monotonic context growth.
  • Empirical Validation: MemAct-RL-14B matches Qwen3-235B accuracy using 49% of the average context length, while learned strategies adapt across models and task complexity.The empirical validation also reports transfer across domains.

2 Related Work

Related work addresses working-memory constraints through compression, selective pruning, and periodic summarization. These approaches seek to fit task-relevant information within fixed context windows.

  • Context Management: Existing methods manage constrained working memory through token-level compression, selective pruning, or periodic summarization.These techniques aim to fit information within fixed context windows.

3 Method

MemAct makes working-memory curation an intrinsic policy action, using Prune&Write to edit context in place and DCPO to train on the resulting non-continuous trajectories. Its MDP formulation assigns addressable memory records and optimizes task success alongside context-efficiency constraints.

  • 3.1 Operational Overview: MemAct integrates context management into the policy action space, allowing agents to autonomously curate working memory through in-place Prune&Write operations.The policy can select task or memory actions, prune indexed historical turns, synthesize memory content, and retain the action record for future updates.
  • 3.2 MDP Formulation: The MDP represents working memory as uniquely addressable interaction records, with memory actions specifying target IDs to remove and generated content to preserve reasoning continuity.Task actions append new interaction records, while memory actions filter selected IDs and append the resulting memory record.
  • 3.3.1 Trajectory Segmentation: DCPO partitions trajectories at memory-edit points into independent segments whose context prefixes remain fixed, preventing gradients from being computed against historically mismatched states.This segmentation restores locally sequential dependencies while supporting optimization on dynamically edited contexts.
  • 3.3.2 Reward Design: The sparse terminal reward jointly incentivizes successful task completion and resource efficiency by rewarding success and penalizing constraint violations such as exceeding maximum context length.The reward design uses rtask > 0 for success and rpen < 0 for violations.
  • 3.3.3 Reward Attribution and Optimization: The training objective assigns each sampled segment the full trajectory’s normalized advantage because final rewards depend on the collective sequence of memory edits and generations.Segments are sampled from reconstructed trajectories, and the policy uses a clipped GRPO-style surrogate objective.

4 Experiments & Results

MemAct is evaluated on synthetic and public benchmarks for reasoning efficiency, accuracy under context pressure, and generalization to harder tasks. Across these evaluations, learned memory actions improve the accuracy–efficiency trade-off, latency, and robustness over baselines and fixed policies.

  • Main Results: 59.1% multi-objective accuracy lets MemAct-RL-14B outperform Qwen3-235B at 53.1% and Tongyi-DeepResearch at 56.0%.It uses 3,500 average input-context tokens per step and 8.2 × 10^4 total tokens on average.
  • Latency and Efficiency: 40% lower total duration than Search-R1 is achieved by MemAct-RL-7B through compact contexts, sparse updates, prefix-cache reuse, and inline memory actions.MemAct avoids auxiliary inference passes that reprocess contexts for summarization or state evaluation.
  • Ablation Analysis: MemAct maintains higher accuracy than Search-R1 despite averaging 28.9 versus 23.5 tool calls, because proactive actions remove irrelevant history.Search-R1 retains all information and consequently experiences token growth and context noise.
  • Ablation Analysis: Reinforcement learning raises multi-objective accuracy from 0.485 for SFT to 0.591, while fixed five-turn memory updates fall behind MemAct on complex tasks.The comparison indicates that learned timing is preferred over rigid memory schedules for synchronizing pruning with reasoning.
  • Scaling to Complex Tasks: 54.3% accuracy on 8-objective tasks exceeds Search-R1’s 39.3%, despite training MemAct-RL on tasks with at most three objectives.Baselines reach a performance bottleneck beyond four objectives, whereas MemAct generalizes to unseen complexities.
  • Scalability and Generalization: MemAct discovers capacity-aware policies: the 7B model increases action frequency from 2.8 to 3.7 on 8-objective tasks, while the 14B model uses pruning modes near 2 and 6 records.The 14B strategy uses fine-grained pruning during reasoning and coarse-grained pruning after sub-objectives finish.

5 Conclusion

The paper presents MemAct as a learnable approach to autonomous context curation and introduces DCPO to make dynamic context updates compatible with efficient reinforcement-learning training. MemAct-RL-14B achieves a strong accuracy–efficiency trade-off, generalizes across complexity and domains, and discovers capacity-aware memory strategies.

  • Conclusion: MemAct internalizes context curation as explicit policy actions, while DCPO restructures trajectories at memory edits for logically consistent reinforcement-learning optimization.The framework treats autonomous context curation as a scalable building block for long-horizon agentic reasoning.
  • Conclusion: MemAct-RL-14B matches models over 16× larger while reducing average context length by 51% and improving end-to-end inference latency.The reported gains establish a superior accuracy–efficiency Pareto frontier.
  • Conclusion: Learned policies autonomously adapt memory-action intensity to model capacity and maintain a focused reasoning trace across task complexities.The conclusion characterizes these specialized strategies as evidence that context curation can be internalized as a learnable skill.

6 Limitations

MemAct remains limited by reinforcement-learning credit assignment, possible deletion of information needed later, inefficient sampling, and lossy compression. The authors frame these constraints as boundaries for current performance and directions for future extensions.

  • Sparse final-output rewards make it difficult to assign credit to specific memory actions, and long-horizon tasks may involve deleting information that becomes relevant later.
  • Random sampling treats memory operations as equally important and may allocate resources to less informative segments, limiting training efficiency in complex scenarios.
  • Lossy compression creates a trade-off between context length and information density because summarized details cannot be recovered.
  • The current study focuses on memory actions within the standard context window, while future work could add external retrieval or tiered caching for higher-fidelity storage.

A.1 Pseudocode for DCPO

DCPO trains the policy from sampled trajectories by assigning prompt-level advantages, segmenting dynamically edited contexts, and computing policy loss on sampled segments. This restructuring accommodates non-sequential context updates during reinforcement learning.

  • DCPO samples multiple trajectories per prompt, obtains rewards, and computes normalized advantages from their mean and standard deviation.
  • The training loop generates context segments from each trajectory’s memory state, tokenizes them with the associated output, and records action masks and trajectory identifiers.
  • DCPO samples a fixed number of segments from the generated segment pool, computes policy loss, updates the policy, and repeats until convergence.

A.2 SFT Cold-start Data Construction

MemAct uses staged prompting to produce successful cold-start supervision when models do not reliably invoke memory actions. Prompt interventions escalate with context length, and only correct trajectories are retained after removing those interventions.

  • Staged prompting addresses failures such as ignoring the memory tool, repeating ineffective calls, or losing coherence after memory edits.
  • Only trajectories with correct final answers are added to the SFT dataset after injected reminder messages are removed.
  • The protocol lets the model act autonomously below a soft threshold, adds a reminder between thresholds, and explicitly prioritizes memory updating above the hard threshold.

A.3 Datasets Statistics

Training combines HotpotQA and Asearcher data for supervised and reinforcement-learning stages, with training instances capped at three objectives. Evaluation extends to tasks with up to eight objectives to test generalization.

  • The SFT phase uses 930 accurate examples divided into 3,000 training segments, while the RL phase uses 10,240 trajectories.
  • Training data are generated from HotpotQA and Asearcher using the staged prompting protocol.
  • Training samples contain at most three objectives, so gains on tasks with four to eight objectives test generalization beyond memorized training patterns.
  • Table 3 categorizes training instances by the number of reasoning objectives across the SFT and RL phases.

A.4 Additional Results

Additional evaluations show that MemAct remains effective across smaller models, evaluation metrics, optimization backbones, and model families, while its efficiency gains are linked to shorter active contexts and sparse memory edits.

  • Results for 7B Model Variants: MemAct-RL-7B achieves the highest average accuracy of 0.485 among the reported 7B models.The comparison covers tasks with multiple objectives, where accuracy is the average success rate for individual sub-objectives.
  • Analysis of Memory Strategies: Fourteen-billion-parameter models generally produce longer reasoning sequences, whereas 7B models rely more on explicit memory storage and more aggressive pruning.The RL 7B model produces memory records averaging 198.7 tokens on 8-objective tasks, and the higher Pruned Actions values indicate more past actions removed per update.
  • Additional Evaluation with Rule-based F1: Under rule-based F1 evaluation, MemAct-14B consistently outperforms Search-r1-14B across all difficulty levels.The relative method ranking remains consistent with the LLM-Judge evaluation, although absolute F1 values are generally lower.
  • PPO-based Variant of DCPO: The PPO-based DCPO variant achieves competitive performance across all difficulty levels while retaining the same context-segmentation mechanism.The optimization backbone is replaced with PPO, while dynamic-context handling remains unchanged.
  • Additional Results on Qwen3-4B: MemAct improves the Qwen3-4B-2507-Instruct base model after both supervised initialization and reinforcement learning.This evaluates transfer beyond the main Qwen2.5-based setting.
  • KV-Cache and Latency Breakdown: MemAct-RL-14B averages 3,500 active context tokens per step and a total token cost of 8.2×10^4, versus 19.3×10^4 for Search-R1-14B and 16.7×10^4 for Qwen3-235B.The efficiency account attributes the latency advantage mainly to reduced prefill burden, while sparse memory updates preserve locally stable prefixes for cache reuse.
Loading 2510.12635v3…