Source-linked AI summary
MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, Paul Pu Liang
TL;DR
Long-horizon agents must reason over evolving, interdependent interactions without the costs and limits of ever-growing context. MEM1 trains a shared internal state that consolidates memory and reasoning while pruning prior context, and evaluates it in compositional multi-turn tasks. Across reported domains, it achieves competitive or superior performance with substantially lower memory use and inference latency, including beyond its training horizon.
Problem
Long-horizon agents that append all past turns face unbounded context growth, increased computation and memory usage, and limited generalization beyond trained lengths.
Method
MEM1 uses end-to-end reinforcement learning to update a compact internal state that integrates prior memory with new observations and discards obsolete context.
Results
Across internal retrieval QA, open-domain Web QA, and multi-turn shopping, MEM1 rivals leading baselines, reduces memory usage by up to 3.5×, and generalizes from 2-objective training to up to 16 objectives.
Takeaways & Limitations
Reasoning-driven memory consolidation provides a scalable alternative for long-horizon interactive agents by combining efficiency with competitive task performance.
Abstract
from arXiv · showhide
Modern language agents must operate over long-horizon, multi-turn interactions, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to unbounded memory growth, increased computational costs, and degraded reasoning performance on out-of-distribution input lengths. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant memory across long multi-turn tasks. At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning. This state integrates prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. To support training in more realistic and compositional settings, we propose a simple yet effective and scalable approach to constructing multi-turn environments by composing existing datasets into arbitrarily complex task sequences. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on a 16-objective multi-hop QA task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-driven memory consolidation as a scalable alternative to existing solutions for training long-horizon interactive agents, where both efficiency and performance are optimized.
1 Introduction
Long-horizon agents must integrate evolving information across interdependent turns, but appending all prior context causes unbounded resource demands and limits generalization. MEM1 addresses this with a learned consolidated state and evaluates it across compositional multi-turn environments.
- Long-horizon interactions require agents to retrieve information, revise beliefs, and adapt across interdependent questions.
- Appending prior observations, actions, and thoughts produces growing context, increasing inference cost and memory usage while challenging generalization beyond the training horizon.
- MEM1 maintains constant memory by updating a consolidated internal state that combines prior memory with newly obtained information and discards obsolete tool outputs.
- MEM1 trains memory consolidation and reasoning end-to-end with reinforcement learning and verifiable task-success rewards.
- Across internal retrieval QA, open-domain Web QA, and multi-turn shopping, MEM1 rivals leading baselines while reducing memory usage by up to 3.5×.
2 Related Work
Related work spans multi-turn reasoning-and-acting agents, supervised or reinforcement-learning training, and memory systems that append or externally summarize interaction history. These approaches motivate MEM1’s integrated treatment of reasoning and memory.
- ReAct established a reasoning-and-action framework for agents interacting with external environments across multiple turns.
- Agent training commonly uses behavior cloning to imitate expert trajectories or reinforcement learning to optimize policies through rewards.
- Appending observations, thoughts, and actions is simple for short interactions but causes unbounded context growth, linearly scaled inference memory, and redundant information.
3 MEM1
MEM1 treats reasoning as consolidated working memory: each turn updates a compact internal state, generates an environmental query or answer, and prunes prior context. Its masked-trajectory training and compositional multi-objective tasks support policy learning under this bounded-memory regime.
- 3.1 Memory as Part of Reasoning: MEM1 interleaves internal-state reasoning, environment queries, answers, and external information using XML-style tags.
- 3.1 Memory as Part of Reasoning: At each turn, MEM1 produces a new internal state that consolidates prior information before generating either a query or a final answer.
- 3.1 Memory as Part of Reasoning: Previous-turn tags are pruned after consolidation, retaining at most two internal states, two queries, and one information element in context.
- 3.2 Masked Trajectory for Policy Optimization: MEM1 uses masked trajectories to stitch evolving interaction contexts into a coherent sequence for policy optimization.
- 3.2 Masked Trajectory for Policy Optimization: A two-dimensional attention mask restricts each token to memory retained at generation time, keeping policy ratios, advantages, KL penalties, and value estimates valid.
- 3.3 Multi-Objective Task Design: Multi-objective QA composes existing questions into one task requiring multiple searches and integrated answers, increasing reasoning turns and memory demands.
4 Experiments & Results
MEM1 is evaluated across multi-objective QA, WebShop navigation, single-objective Wikipedia QA, and zero-shot online Web-QA. Across these settings, it combines competitive or improved effectiveness with lower memory and inference costs, while emergent traces reveal structured memory and search behaviors.
- Evaluation Setup: MEM1 is trained and evaluated with accuracy, reward, and efficiency metrics across retrieval QA, WebShop, and online web-search environments.The evaluation includes EM, F1, environment reward, Peak Token Usage, Dependency Length, and Inference Time.
- Multi-Objective QA: MEM1 maintains an almost constant peak token count as objectives increase, while other methods scale nearly linearly.At higher objective counts, baseline performance degrades and some models collapse.
- Multi-Objective QA: 27.1% of Qwen2.5-14B-Instruct’s peak tokens and 29.3% of its total inference time are required by MEM1 on the 16-objective task.MEM1 eventually surpasses the 14B model despite having half its parameter count.
- WebShop: MEM1 outperforms Agent-Flan, Agent-R, and AgentLM on WebShop with 2.8× better Peak Token Usage, 1.9× better Dependency, and 1.5× better Inference Time.It also surpasses AgentLM-13B and performs significantly better than GPT-4o on the reported WebShop tasks.
- Single-Objective QA: On single-objective Wikipedia QA, MEM1 achieves the highest EM, comparable F1 to Qwen2.5-14B-Instruct, and superior efficiency across all three efficiency metrics.The same agent was trained only on a 2-objective task; the reported efficiency is attributed to compact internal-state memory consolidation.
- Zero-Shot Transfer: Zero-shot transfer to unseen online Web-QA preserves comparable effectiveness while improving efficiency.The online environment returns search titles, snippets, and URLs through an API.
- Emergent Behaviors: Interaction traces show separate memory management for concurrent questions, focus shifts when progress stalls, and interleaved reasoning that guides subsequent searches.The traces also show extraction of useful information from prior search results to formulate queries addressing current information gaps.
5 Conclusion, Limitations, and Future Work
MEM1 integrates long-horizon reasoning and memory consolidation into a unified state, achieving competitive QA and web-navigation performance with reduced memory usage and inference latency. Its current scope depends on environments with well-defined, verifiable rewards, leaving open-ended settings for future work.
- Conclusion: MEM1 unifies inference-time reasoning and memory consolidation in a consolidated internal state for long-horizon agents.The framework is reported to address prompt-growth scalability while retaining competitive benchmark performance.
- Conclusion: MEM1 achieves competitive performance across QA and web navigation with substantially reduced memory usage and inference latency.This is the paper’s reported overall conclusion across the evaluated benchmark families.
- Limitations and Future Work: MEM1 assumes environments provide well-defined and verifiable rewards, which excludes many open-ended tasks with ambiguous or noisy reward structures.Future work is proposed for sparse, delayed, or implicit reward signals.
A.1 Computing Resources and Training Details
The appendix specifies the hardware, optimization settings, retrieval services, and prompt protocols used for MEM1 training and evaluation. The task prompts require cumulative reasoning and memory updates followed by either searches, environment actions, or concise answers.
- Computing Resources and Training: MEM1 training uses 4 H100 or H200 GPUs with veRL for RL and Swift for SFT.RL uses batch and mini-batch size 64, actor learning rate 10^-6, critic learning rate 10^-5, and 50 warmup steps.
- Computing Resources and Training: Evaluations run on a single H200 GPU through vLLM with automatic prefix caching.Models are served as API services.
- Retrieval Services: Local Wiki retrieval uses Faiss-GPU with an E5 Base model over a Wikipedia 2018 dump, returning 3 passages per retrieval.The retrieval count is fixed for comparison with other methods.
- Retrieval Services: Online searches use Serper API and return the top 10 results with titles, snippets, and URLs.The agent is not asked to retrieve the contents of specific webpages.
- QA Prompt Protocol: Multi-objective QA prompts require iterative reasoning, cumulative summaries, one search at a time, and concise semicolon-separated final answers.The persistent summary retains essential information from prior reasoning and retrieved information.
- Single-Objective QA and WebShop Protocols: Single-objective QA prompts alternate <think> with either <search> or <answer>, while WebShop prompts require valid stepwise search and click actions.WebShop actions include product search, navigation, attribute inspection, and buy-now selection.
- Single-Objective QA and WebShop Protocols: WebShop agents navigate page states by searching, clicking products or attributes, paging through results, and selecting buy now when available.Only valid action formats such as search[...] and click[...] are permitted.
A.4.1 Metrics
MEM1 evaluates answer accuracy and interaction efficiency using task-specific rewards and metrics covering correctness, memory, dependency, and latency. Its rollout procedure resets retained context each turn while preserving the current turn’s sequence.
- Accuracy metrics: Exact match serves as both the QA reinforcement-learning reward and the final-output evaluation metric.Answers are extracted from <answer> and </answer>; multi-objective responses must answer each question in semicolon-separated form.
- Accuracy metrics: F1 score measures the harmonic mean of precision and recall over tokenized predicted and reference answers.For multiple ground truths, the maximum F1 is selected; multi-objective F1 sums sub-question scores.
- Efficiency metrics: Peak token usage is the maximum token count in any trajectory sequence and proxies inference-time memory requirements.The system prompt is excluded, and tokenization uses GPT-4o-mini’s tokenizer.
- Efficiency metrics: Dependency length measures the total historical-token dependence of generated tokens across interaction steps.MEM1 consolidates prior-step prefixes into a new internal state rather than continuously accumulating them; system-prompt tokens are excluded.
- Efficiency metrics: Inference time is the total elapsed time needed to generate a complete output trajectory.Measurements use one H200 GPU, 10 concurrent threads, and vLLM with automatic prefix caching.
- Rollout procedure: MEM1 uses PPO-based rollouts that generate tokens until query, answer, or end-of-sequence markers, removing previous context after each turn.The rollout is initialized with a task prompt, policy model, world model, and maximum turn count.
A.6 MEM1 on Webshop Training Details
WebShop training uses MEM1’s rollout and policy-update pipeline with an environment-specific prompt and reward signal. Unlike QA, it relies on WebShop’s built-in state reward rather than exact match.
- Training setup: MEM1 applies the same rollout pipeline and policy-update mechanism to WebShop while tailoring the prompt to memory consolidation in that environment.The prompt includes instructions specific to WebShop while retaining the core memory-consolidation objective.
- Reward design: WebShop training uses its built-in reward function for each environment state instead of QA’s exact-match reward.The environment supplies reward signals corresponding to the agent’s states.
A.7 Additional Discussion on the Attention Matrix Design.
MEM1’s attention-mask design approximates the original trajectory while avoiding costly duplication of information-state tokens. The authors report that this efficiency-oriented deviation has no significant experimental impact.
- Attention design: The attention-matrix modification cannot fully recover original-trajectory attention because memory consolidation changes position ids.Each information-state token can have separate positions for the previous and next turns.
- Attention design: Duplicating each information-state token with separate position ids would recover the original attention but substantially lengthen training trajectories.Prior trajectory-compression methods adjusted position ids because their sequences shared prefixes; MEM1’s consolidated prefixes differ.
- Implementation trade-off: For training efficiency, MEM1 assigns each information-state token the previous-trajectory position id instead of duplicating it.The authors characterize this as a slight deviation from the ideal implementation.
- Implementation trade-off: The position-id approximation is described as equivalent to adding whitespace and having no significant impact on experimental results.This claim qualifies the practical effect of deviating from the ideal attention reconstruction.
- Broader motivation: MEM1 targets scalable agents that sustain long, goal-directed interactions in dynamic environments through memory consolidation.The discussion connects this direction to applications including scientific research, legal analysis, personalized education, and digital customer service.
C Training Trajectory Analysis of MEM1
MEM1’s training trajectory passes through exploration, format acquisition, a shortcut-driven local optimum, renewed information gathering, and policy consolidation. The reported dynamics show that reward improvements do not always indicate more complete answers.
- Training trajectory: During the first 50 training steps, low reward, high entropy loss, and roughly 0.55 valid-action ratio indicate weak task proficiency and unreliable formatting.The agent has not yet reliably learned the required query-and-answer structure.
- Training trajectory: Format acquisition raises valid-action ratio and reward, linking structural consistency with improved task success.This phase occurs before the later shift toward fewer searches.
- Shortcut behavior: Around step 150, valid searches fall while reward rises, suggesting exploitation of the reward structure through formatting compliance over content completeness.The analysis characterizes this as short-horizon optimization.
- Shortcut behavior: Between steps 150 and 200, the agent reaches a local optimum with valid but under-informed answers as reward growth slows and entropy flattens.Valid-action ratios continue increasing while search counts remain low.
- Policy refinement: After step 200, rising valid searches signal renewed information acquisition, sharper reward growth, and better balance between formatting and search.After step 250, sharply falling entropy loss marks policy consolidation into a more deterministic, high-reward policy.
D Analysis on Implementation Details
The implementation analysis compares reinforcement learning with supervised fine-tuning and examines format rewards. Format rewards accelerate convergence but reduce final task performance, possibly by constraining exploration of effective reasoning strategies.
- RL versus SFT: MEM1-QA trained with supervised fine-tuning consistently underperforms its reinforcement-learning counterpart on increasing multi-turn QA tasks.Both models are trained on the 2-objective QA task, while the SFT model is additionally trained on 1-objective and 3-objective tasks.
- Format reward: Format reward produces faster training convergence but worse final performance.The comparison is shown in Figure 7 for MEM1 trained with and without format reward.
- Format reward: Format reward enforces <IS>, <query>, and <answer> tags, terminating malformed turns with a -1 penalty.The reward was tested as an alternative training signal for reasoning and multi-turn agents.
- Format reward: 0.466 exact match with format reward versus 0.709 with outcome-based reward on the same 2-objective QA test set.The format-constrained agent also has an average peak of 514.9 tokens, compared with 640 tokens for outcome-reward-trained MEM1.
- Format reward: The authors hypothesize that format reward accelerates structural learning while constraining exploration of effective reasoning strategies.They further associate this trade-off with shorter, syntactically valid responses and less effective internal state representations.