Source-linked AI summary
REMORY: Learning Residual Memory for Context Compaction
Hanchen Xia, Baoyou Chen, Yutang Ge, Naihao Deng, Senqiao Yang, Zilong Dong, Weihao Yuan, Siyu Zhu
TL;DR
Long-horizon agents must compact history, but summaries may not support every later decision. REMORY adds bounded, summary-conditioned soft memory tokens to help a frozen LLM approximate full-history continuation. It improves source attribution and agent benchmark performance while using a small fraction of raw-context positions on SummHay.
Problem
Long-horizon agents must compact history to fit finite context windows, yet a textual summary may not support every subsequent decision as earlier context becomes relevant in new ways.
Source: §1 · p. 1Method
REMORY generates bounded soft memory tokens conditioned on the summary and appends them after it to help a frozen LLM approximate full-history continuation.
Source: §1 · p. 2, §2.1 · p. 3Results
Both evaluated agents gain across long-horizon benchmarks; on SummHay, summary plus memory uses 5.2% of raw-context positions and approaches its joint score with higher citation F1 and lower coverage.
Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 5Takeaways & Limitations
The summary provides a readable account, soft tokens carry additional predictive information, and retrieval supplies exact records when needed.
Source: §5 · p. 7Generation-cost estimates exclude source-feature encoding, memory-network execution, and tools, so they measure token-priced generation rather than total execution cost.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.
1 Introduction
Long-horizon agents must compact growing interaction histories, but a textual checkpoint may not support every later decision. REMORY supplements the summary with learned soft tokens to help a frozen LLM approximate full-history continuation.
1 Introduction
- Agent context grows with interaction and drops at compaction, motivating a learned memory paired with the explicit checkpoint.Figure 1 distinguishes within-actor comparisons with and without residual memory from published frontier references using different harnesses and protocols.Source: §1 · pp. 1–2, Figure 1 · p. 2
- A textual checkpoint may preserve the main thread yet fail to support every subsequent decision as the relevance of earlier context changes.The authors do not assume that needed information beyond the summary can be identified or enumerated in advance.Source: §1 · p. 1
- REMORY generates bounded soft memory tokens conditioned on the summary and appends them after it to help a frozen LLM approximate full-history continuation.The summary remains a readable checkpoint, while exact records remain accessible through ordinary archive and file-retrieval tools.Source: §1 · p. 2
- On SummHay, residual memory improves source attribution at nearly unchanged insight coverage, while both evaluated agents gain on long-horizon benchmarks and have fewer repeated tool outputs and errors.The paper reports these tool-output and error reductions on BrowseComp and Terminal-Bench 2.1.Source: §1 · p. 2
2 Residual Memory
REMORY supplements a generated summary with continuous memory tokens and trains this interface through a frozen actor. The method combines hierarchical memory construction with reconstruction and summary-conditioned continuation training.
2 Residual Memory
- Figure 3 depicts summary-conditioned memory construction and teacher–student top-1 agreement diagnostics for reconstruction and residual-training stages.The residual-training panel compares reconstruction initialization with random initialization under the same objective and data order; its curves are optimization diagnostics.Source: Figure 3 · p. 3
2.1 A memory supplement to the summary
- At each compaction, the network conditions on the new summary to construct memory from the preceding summary, prior memory, and intervening interaction; the actor reads the concatenated sequence.Hierarchical compression produces 64 soft tokens per block of up to 1,024 source positions and bounds final memory at 4,096 slots, while preserving source order.Source: §2.1 · p. 3
2.2 Learning through a frozen actor
- Table 1 evaluates SummHay on the same 92 queries and compares residual memory with Summary++ text using the residual memory's additional-position budget.All queries, including generation and formatting failures, are retained.Source: Table 1 · p. 4
- Training first reconstructs source text from soft memory, then trains summary-conditioned memory to predict the same future assistant response as the full-history teacher.In Stage II the memory encoder does not receive the future response, and only memory parameters are updated through the frozen actor.Source: §2.2 · p. 4
3 Experiments
Experiments test residual memory on source attribution, end-to-end agent benchmarks, post-compaction actions, and case studies. Results show gains across benchmark settings, with attribution improving using a small fraction of full-context input positions.
3.1 Source attribution after compaction
- Citation F1 rises by 4.04 points and joint score by 3.55, with a coverage change of 0.40; on 553 shared covered insights, citation F1 rises from 57.31 to 61.41.Summary plus memory uses 5,024 input positions, 5.2% of raw context, and approaches its joint score with lower coverage and higher citation F1; the equal-budget text supplement stays close to summary alone.Source: §3.1 · pp. 4–5
3.2 End-to-end agent performance
- Repeated tool outputs fall by 17.9–29.8% and errors by 25.3–49.7% across four actor–benchmark pairs.Repeats exclude planning and polling; errors count failed commands or explicit tool errors.Source: Table 3 · p. 5, §3.2 · pp. 5–6
- Qwen gains 9.8 points on AutomationBench and 7.6 on JobBench; on Terminal-Bench it gains 4.5 and 3.0 on BrowseComp, while GLM gains 3.3 and 4.1, respectively.The within-actor comparisons hold the evaluation setup fixed; the improvements also hold for the stronger actor with a larger context window.Source: §3.2 · p. 5, Table 2 · p. 5, Table 3 · p. 5
3.3 The first action after compaction
- Residual actions receive 345 wins (70.7%), 21 ties (4.3%), and 122 losses (25.0%) in paired judgments of 488 next actions.With the checkpoint fixed, the preference concerns proposed first post-compaction actions, not outcomes of the unexecuted counterfactuals.Source: §3.3 · p. 6
3.4 Case studies
- In Live Trading, settled ROI is −11.86% without residual memory and +5.11% with it, but the two trajectories differ in market selection and timing.Compactions fall from 56 to 45 alongside fewer responses; per 1,000 responses, they rise from 10.5 to 14.9, while lower calls and smaller contexts explain lower generation spend in this record.Source: §3.4 · p. 6, Figure 4 · p. 6
- In cell segmentation, the actor restores deleted helper functions and passes all nine verifier tests after rereading both files.The repair illustrates complementary roles for summary, memory, and retrieval, but does not establish which omitted details came from memory.Source: §3.4 · pp. 6–7, Figure 5 · p. 7
4 Related Work
Prior agent-memory approaches combine summaries with retrieval or learn continuous representations of text. REMORY adds a summary-conditioned continuous supplement trained to predict a frozen actor’s continuation.
4 Related Work
- ReadAgent pairs episode gists with original passages, while MemGPT separates working and archival memory (Lee et al., 2024; Packer et al., 2023).ReSum uses periodic summaries for long-horizon search, while AgentFold learns multi-scale context management within agent trajectories (Wu et al., 2025; Ye et al., 2025).Source: §4 · p. 7
- REMORY preserves the textual checkpoint and retrieval interface while learning an additional continuous representation conditioned on the checkpoint.Its objective applies distribution matching to the information gap left by a summary, training the supplement through a frozen actor’s continuation predictions (Wingate et al., 2022).Source: §4 · p. 7
- AutoCompressors, Gist Tokens, and ICAE learn continuous representations of longer text (Chevalier et al., 2023; Mu et al., 2023; Ge et al., 2024).LCLMs scale encoder–decoder compression and support selective expansion of compressed segments (Li et al., 2026a), while recurrence and memory tokens retain information across segments (Dai et al., 2019; Bulatov et al., 2022).Source: §4 · p. 7
5 Conclusion
REMORY supplements a readable summary with bounded predictive memory tokens and retrieval for exact records. Its training interface leaves the actor unchanged, and reported results support the usefulness of the residual formulation.
5 Conclusion
- REMORY learns a bounded memory supplement for contextual compaction, with the summary providing a readable account and soft tokens carrying additional predictive information.Retrieval supplies exact records when needed.Source: §5 · p. 7
- Reconstruction followed by summary-conditioned continuation trains the interface while leaving the actor unchanged.Source: §5 · p. 7
- Improved source attribution, higher agent scores, and preferred next actions support the usefulness of the residual formulation.Source: §5 · p. 7