Source-linked AI summary
NextMem: Towards Latent Factual Memory for LLM-based Agents
Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua
TL;DR
Factual memory is foundational for LLM-based agents, but textual and parametric approaches carry context, indexing, forgetting, and storage-cost limitations. NextMem introduces a latent factual-memory framework using an autoregressive autoencoder, two-stage training, and quantization. Experiments report high-fidelity reconstruction, robust retrieval, and significant storage reduction, while identifying a trade-off between reconstruction accuracy and instruction-following capability.
Problem
Textual factual memory incurs context-length and indexing overhead, while parametric memory faces catastrophic forgetting and high storage costs.
Method
NextMem uses an autoregressive autoencoder, two-stage training, and quantization to encode and reconstruct latent factual memory efficiently.
Results
NextMem achieves high-fidelity reconstruction, robust retrieval, and significant storage reduction across experiments.
Takeaways & Limitations
NextMem provides a scalable foundation for factual memory in LLM-based agents.
Takeaways & Limitations
The reported results identify a trade-off between reconstruction accuracy and instruction-following capability.
Abstract
from arXiv · showhide
Memory is critical for LLM-based agents to preserve past observations for future decision-making, where factual memory serves as its foundational part. However, existing approaches to constructing factual memory face several limitations. Textual methods impose heavy context and indexing burdens, while parametric methods suffer from catastrophic forgetting and high costs. To address these challenges, we introduce NextMem, a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory while ensuring accurate reconstruction. For better optimization, we propose a two-stage training process, including autoregressive reconstruction alignment and progressive latent substitution. We also incorporate quantization to reduce storage overhead. Extensive experiments demonstrate that NextMem achieves superior performance, and excels in retrieval, robustness, and extensibility properties. We release our code and model checkpoints at https://github.com/nuster1128/NextMem.
1 Introduction
Factual memory preserves observed details, but textual and parametric approaches impose substantial burdens or risks. NextMem addresses these limitations with reversible latent representations, two-stage training, and quantization.
- Factual memory preserves observed facts, whereas task-oriented memory extracts task-specific information from original observations.
- Textual memory increases context length and indexing overhead when storing and using many detailed facts.
- Parametric memory can suffer catastrophic forgetting and high costs when storing detailed facts accurately.
- NextMem uses reversible latent representations to shorten textual memory while accurately reconstructing the original information.
- NextMem combines autoregressive reconstruction alignment, progressive latent substitution, and quantization to reduce storage cost while preserving accuracy.
2 Preliminaries
Agent memory is framed as encoding historical information and decoding it to augment LLM inference. The preliminaries distinguish textual, parametric, and latent representations within this unified perspective.
- 2.1 Memory in LLM-based Agents: Agents encode historical information into memory and later decode it to augment LLM inference.
- 2.1 Memory in LLM-based Agents: The encoding-decoding view unifies memory forms beyond traditional storage, retrieval, and utilization.
- 2.2 Representation of Agent Memory: Textual memory stores text, uses indexing for retrieval, and integrates relevant context into LLM inference through prompts.
- 2.2 Representation of Agent Memory: Parametric memory incorporates information by modifying model parameters and adds cumulative modifications to the base parameters at inference.
- 2.2 Representation of Agent Memory: Latent memory transforms textual information into latent representations that can reduce sequence length and inference latency.
3 Methods
NextMem uses an autoregressive autoencoder to encode text into latent representations compatible with LLM inputs and reconstruct the original text. Its two-stage training aligns autoregressive reconstruction before progressively replacing text blocks with latent representations, while quantization reduces storage overhead.
- Autoregressive Autoencoder: NextMem uses a causal-language-model autoregressive autoencoder with separate encoder and decoder weight sets to transform text into LLM-compatible latent representations.The architecture includes embedding, Transformer, and language-modeling components, with a special [SoD] token initiating transformation.
- Autoregressive Autoencoder: The encoder generates latent embeddings iteratively by extracting the final hidden state and appending each previously generated embedding to subsequent inputs.The final latent representation concatenates all generated latent embeddings.
- Autoregressive Autoencoder: The decoder concatenates the latent representation with a suffix embedding and autoregressively predicts tokens over the vocabulary.This produces an output sequence from the latent memory representation.
- Autoregressive Reconstruction Alignment: Stage one trains autoregressive reconstruction alignment using self-supervised sequences that contain the original text, a transformation token, and reconstruction targets.The causal language model is fine-tuned by maximizing the likelihood of the target sequence while ignoring designated positions.
- Progressive Latent Substitution: Stage two progressively substitutes original text blocks with latent representations, forcing the decoder to recover missing textual information while only optimizing the encoder against a frozen decoder.Encoder initialization proceeds from the preceding progressive step, and stop-gradient detachment reduces recurrent backpropagation cost.
- Latent Memory Quantization: NF4 quantization maps high-precision latent representations to 4-bit indices with per-feature scales, then reconstructs approximations by retrieving codebook values and rescaling them.Indices are stored as 4-bit unsigned integers and scales use FP8 format; the framework also uses shared backbone parameters with LoRA adapters.
4 Experiments
NextMem is evaluated across factual reconstruction, contextual generation, retrieval, ablations, compression, robustness, semantic assignment, and optimization behavior. Across these analyses, it generally preserves information effectively, supports retrieval, tolerates perturbations, and benefits from its proposed components.
- Experimental Setup: Experiments evaluate latent memory across storage, utilization, retrieval, compression, robustness, forgetting, and other properties.The primary tasks are factual reconstruction, contextual generation, and dense passage retrieval.
- Experimental Setup: The evaluation uses SQuAD, HotpotQA, RACE, and LoCoMo, with NextMem-Dense and quantized NextMem-Sparse generating 15 latent tokens.Baselines include DeepSeek-OCR, ICAE, DyPRAG, Textual Memory, and BGE.
- Major Performances: NextMem methods significantly outperform baselines on factual reconstruction, while NextMem-Sparse remains highly competitive after quantization.NextMem-Dense achieves the highest scores in most scenarios, whereas DyPRAG and DeepSeek-OCR show limited reconstruction capabilities.
- Major Performances: NextMem outperforms all baselines in decompressed contextual generation, although ICAE performs better in direct compressed inference.The results indicate a trade-off between reconstruction accuracy and instruction-following capability.
- Major Performances: NextMem substantially improves dense passage retrieval and unifies memory storage and retrieval in one latent representation.This unification reduces architectural complexity according to the reported results.
- Ablation Studies: Removing any ablation component degrades performance, with progressive latent substitution identified as the most critical component.Removing quantization scaling also causes a drastic decline for the sparse model.
- Compression and Robustness: NextMem degrades more slowly as input length increases and preserves high semantic integrity beyond the 240-token training length.The model shows a slower and more graceful performance decay than other models, despite slight dips on shorter sequences.
- Compression and Robustness: NextMem remains stable under moderate Gaussian noise with σ ≤ 0.8, while NF4 quantization causes negligible performance loss.The model still provides meaningful information under high perturbations.
5 Related Works
Related work frames agent memory as either textual or parametric, including systems for context management and methods that inject knowledge into model parameters.
- Related Works: Textual memory systems store conversations or manage context, while parametric methods modify model parameters to inject knowledge.The related work discusses MemoryBank, MemGPT, and model editing approaches.
6 Conclusion
NextMem is an autoregressive autoencoder framework for efficient latent factual memory. Its two-stage training and NF4 quantization support high-fidelity reconstruction, robust retrieval, and substantial storage reduction.
- NextMem uses an autoregressive autoencoder framework for efficient latent factual memory.
- Its two-stage training process and NF4 quantization achieve high-fidelity reconstruction, robust retrieval, and significant storage reduction.
- Experiments validate NextMem as a scalable and effective foundation for memory in LLM-based agents.
A.1 Failure Cases on Reconstruction
The reconstruction experiments show that early latent-memory designs failed to preserve or reconstruct text reliably. Progressive latent substitution and direct latent hidden-state generation produced stable training and reliable encoding and decoding.
- Version 1: Version 1 reduced loss from 2.18 to 1.24 but learned to copy reference text rather than produce the desired latent representation.
- Version 2: Version 2 reduced loss from 2.4 to 1.7 but generated meaningless text, with similar latent representations and inadequately learned special tokens.
- Version 3: Version 3 introduced autoregressive reconstruction alignment followed by latent-space optimization to encode explicit text into latent representations and decode it back.
- Version 4/Final: Version 4 replaced one-time optimization with progressive latent substitution and switched to direct latent hidden-state generation, producing stable loss decline and reliable encoding and decoding.
A.2 Failure Cases on Sparsity
Several sparsification strategies failed to reduce latent-memory storage while preserving accurate representations. Failures included non-convergence, representation collapse, high reconstruction error, incoherent token selection, and numerical instability.
- Version 5: MoE-inspired sparsification left training loss stagnant at approximately 3.3 and failed to converge further.
- Version 6: RQ-VAE reduced training loss to approximately 1.0 but caused representation collapse and decoded outputs that overfit memorized training data.
- Version 7: OMP produced significant residuals, indicating inaccurate approximation of the original latent signals.
- Version 8: Explicit projection selected isolated original tokens in a disjointed pattern rather than cohesive compressed semantic information.
- Version 9: Reparameterization caused numerical instability, with training loss fluctuating between 1.0 and 2.0 and resulting in poor performance.
B.1 Dataset Preparation
The datasets are standardized around questions, answers, and references, then processed differently for reconstruction, contextual generation, and dense passage retrieval. These procedures vary context construction and preserve task-specific evidence or hit-list information for evaluation.
- Dataset standardization: Each dataset is standardized into three primary fields: question, answer, and reference.
- Dataset-specific processing: HotpotQA uses easy samples and supporting sentences, while RACE aggregates high-level data and treats the article context as the reference.
- Task 1: Factual Reconstruction: Reconstruction evaluation uses strided sampling and stochastic truncation to create normalized reference snippets with diversified lengths.
- Task 2: Contextual Generation: Contextual generation evaluation uses the processed testing data.
- Task 3: Dense Passage Retrieval: Dense passage retrieval evaluation expands references with non-essential or session-wide context and records the indices of required references in a hit list.
B.2 Model and Training Configuration
The experiments use distinct model configurations for NextMem and its comparison systems. NextMem relies on an autoregressive setup, while baselines are tailored to contextual utilization or retrieval tasks.
- NextMem uses Qwen3-8B as its autoregressive autoencoder backbone with 1024-token encoding and output limits.Training uses 15 progressive steps with block size 16 and LoRA fine-tuning.
- DeepSeek-OCR renders text as 240 × 240 pixel images corresponding to 16 latent tokens for decoding.The rendering uses PIL, black text on white backgrounds, Times New Roman, padding, and automatic line wrapping.
- ICAE generates 128 latent tokens from a Mistral-7B-Instruct checkpoint using LoRA with rank r = 512 and dropout 0.05.
- DyPRAG uses a Llama-3-8B-based checkpoint to generate LoRA adapters with rank r = 2, α = 32, and projector p = 32.
- Textual Memory is evaluated only in Task 2 for contextual utilization using a reference-based answering prompt.
- BGE-M3 is evaluated only in Task 3 as a dense passage retrieval comparison model through SentenceTransformers.
C Influence of Block Size
Performance depends on both latent length and substitution block size, while simulated memory decay remains mild initially but eventually degrades all evaluation metrics.
- Performance increases with latent length L because greater memory capacity supports more nuanced information storage.
- Performance peaks at B = 16, while both small and large block sizes are suboptimal for reconstruction or optimization.At B = 8, long paragraphs are difficult to reconstruct at a fixed latent length; large B values hinder optimization.
- All evaluation metrics remain relatively stable during the initial forgetting steps t ≤4.
- As t increases further, all scores decline sharply and consistently as latent representations converge toward the average embedding.The decay process uses α(t) = a^t, where a controls the forgetting rate.
E.1 Case Study on Paragraphs with Various Lengths
NextMem reconstructs detailed text nearly perfectly below its length limit, preserves core content near and beyond the limit, and loses semantic fidelity as noise increases.
- Below the length limit, reconstruction preserves semantic content, syntax, and specific entities with near-perfect fidelity.The case study reports only negligible variations, including capitalization changes.
- At the length limit, the model preserves the narrative structure and most profile attributes but makes minor associative substitutions.Examples include replacing Tesla Model 3 with Model S and violin with bass.
- Beyond the length limit, reconstruction retains major life events and persona coherence while blending some secondary details.Professional background, academic honors, and upcoming travel plans remain intact despite higher information density.
- Reconstruction fidelity decreases as Gaussian noise σ increases, progressing from near-identical text at σ = 0.4 to structural breakdown at σ = 1.2.Intermediate noise produces factual substitutions and semantic drift before substantial consistency loss.