Memorilla: Latent Semantic Memory for LLMs

• At 16 memory tokens, MEMORILLA has the highest mean score across eight benchmarks on Figure 1’s Pareto frontier, using 394× fewer document tokens than full context.Source: Figure 1 · p. 2
• Existing systems either place selected text in context, form latent tokens without the query, compress a single retrieved document, or depend on model-specific memory; the paper identifies query-relevant retrieval from document embeddings into a frozen LLM’s input space as missing.The motivating challenge is to give a frozen LLM access to an entire, growing knowledge collection while keeping its context short.Source: §1 · pp. 1–2
• MEMORILLA encodes each document once into a persistent store, then uses a trainable module to read the collection for each query and return 16 latent tokens to a frozen LLM.The tokens are query-conditioned and do not require documents to enter the context as text.Source: §1 · p. 2
• On LaMP-4, MEMORILLA reaches 18.15 ROUGE-L versus 14.21 for full context; on FactKG, it reaches 84.33% versus 80.43%.The FactKG result uses 108× fewer document tokens; TriviaQA reaches the highest exact match, 57.00, with 743× fewer document tokens than full context.Source: §1 · pp. 2–3
• In TextWorld, MEMORILLA completes 37.5% of held-out games in at least one of three attempts, versus 25.0% for the same agent given its full history in context.The authors report that it improves the agent’s ability to escape navigation loops and execute action sequences.Source: §1 · p. 3

2 MEMORILLA

• For each query, MEMORILLA reads the document store and returns exactly K soft memory tokens for the frozen LLM.The system separates persistent document storage from query-conditioned retrieval.Source: §2 · p. 3, §2.2 · p. 3

2.1 ENCODING AND PERSISTENT STORAGE

• A frozen encoder maps document chunks to query-independent embeddings that are stored once; only added or changed documents need re-encoding.A separate frozen query encoder produces an embedding for the current query, which is not stored as long-term memory.Source: §2.1 · p. 3

2.2 QUERY-CONDITIONED RETRIEVAL (fMEM)

• Learned memory slots are initialized across collections and queries, conditioned on the query, then refined by attending to every stored document embedding.This implements a continuous, query-conditioned read rather than discrete top-k selection.Source: §2.2 · p. 4

2.3 MEMORY INJECTION INTO THE FROZEN LLM

• A learned projection maps the refined slots to the LLM input-embedding dimension, where they join the query in the frozen LLM’s input sequence.The LLM’s layers and attention mechanism remain unchanged.Source: §2.2 · p. 4, §2.3 · p. 4

2.4 SUPERVISED TRAINING OBJECTIVE

• Training minimizes the negative log-likelihood of target answer tokens, applying the loss only at answer positions.Prompt and memory positions are excluded from the loss.Source: §2.4 · p. 4

2.5 MEMORILLA WITH QWEN3

• The Qwen3 instantiation uses a frozen Qwen3-Embedding-4B encoder, a frozen Qwen3-8B LLM, and K = 16 memory tokens.The configuration has 194,123,776 trainable parameters.Source: §2.5 · p. 4

2.6 TRAINING STAGES

• A three-stage curriculum moves from Wikipedia paraphrase reconstruction to diverse language tasks, then to the evaluation benchmarks.The final stage uses one multi-task checkpoint for six benchmarks, with additional benchmark-specific fine-tuning for PersonaMem-v2 and PersonalizationV4.Source: §2.6 · p. 5

3.1 BASELINES

• Comparisons cover cosine-similarity RAG, Mem0, MemoRAG, xRAG, ComMer via reported results, and LLMLingua-2; author-run baselines use frozen Qwen3-Embedding-4B encoders and frozen Qwen3-8B decoders.Mem0 and MemoRAG additionally use a frozen Qwen3-4B-Instruct-2507 model to build their memories.Source: §3.1 · p. 5

3.2 BENCHMARKS

• The eight supervised benchmarks span personalization, long-document question answering, and knowledge-intensive question answering.They test integration of user history, compression of long documents for QA, and factual grounding from structured or semi-structured evidence.Source: §3.2 · p. 5

3.3 EVALUATION

• Evaluation uses accuracy, exact match, token-level F1, ROUGE, BLEU, F1, and METEOR as appropriate to each task.Supervised evaluations use temperature 0, and the decoder context is capped at 32,768 tokens, with YaRN enabled when full-context inputs exceed the limit.Source: §3.3 · p. 5
• MEMORILLA lies on the Pareto frontier for every benchmark, leads on five, and ranks second on NarrativeQA and PV4.Figure 3 plots primary-metric score against mean document tokens per query using a frozen Qwen3-8B decoder.Source: Figure 3 · p. 6

Overall results

• MEMORILLA outperforms all baselines on average while using 394×, 113×, and 49× fewer document tokens than full context, RAG-50, and MemoRAG, respectively.It leads on five benchmarks and ranks second on NarrativeQA and PV4.Source: §4 · p. 6

Relational evidence

• On FactKG, MEMORILLA reaches 84.33% against 80.43% for full context, 80.38% for MemoRAG, and 78.98% for xRAG.It uses 108× fewer tokens than full context and 36× fewer than MemoRAG.Source: §4 · p. 6

Long-document and personalization QA

• On TriviaQA, MEMORILLA reaches 57.00 EM versus 56.60 for MemoRAG and 52.60 for full context; on PMv2 short-answer, it reaches 44.60% versus xRAG's 30.90%.It also improves LaMP-7 from the closed-book model's 43.13 to 48.37 ROUGE-L, and reaches 18.15 ROUGE-L on LaMP-4 with 474× fewer tokens than full context.Source: §4 · p. 6

Where memory does not help

• On NarrativeQA, MemoRAG leads in ROUGE-1 and ROUGE-L with 21.80 and 20.51 versus MEMORILLA's 19.68 and 19.31; on PV4, it leads 67.88% to 66.65%.On PubMedQA, LLMLingua-2 reaches 69.00% and four other baselines reach 67.00%, versus 59.00% for MEMORILLA.Source: §4 · p. 6

Full context is not an upper bound

• Full context trails MEMORILLA on every benchmark except PubMedQA and on 20 of 21 metrics, while RAG-50 exceeds full context on 15 metrics.Supplying the entire collection therefore does not consistently improve performance.Source: §4 · p. 6

Per-question analyses

• MEMORILLA favors NarrativeQA's conceptual questions, while RAG-50 favors entity-recall questions; MEMORILLA's advantage grows with entity count and remains stable as more facts are composed.The union of MEMORILLA and RAG-50 answers a larger share of questions than either method alone on every benchmark.Source: Figure 4 · p. 7, Figure 5 · p. 7

Multi-entity relational reasoning

• MEMORILLA's FactKG advantage over RAG-50 grows from +4.23pp at one entity to +7.59pp at four or more; RAG-50 falls from 78.73 to 51.75 as composed facts increase from one to four, while MEMORILLA stays in the 80–89 range.The passage attributes this pattern to holistic relational structure in compression and retrieval degradation across multi-document reasoning chains.Source: §5 · p. 7

Complementary answers

• The union of MEMORILLA and RAG-50 is an oracle upper bound that could yield gains over either method alone, motivating hybrid retrieval and latent-memory systems as future work.On FactKG, 15.9% of questions are answered by MEMORILLA alone versus 10.0% by RAG-50 alone; on PMv2 short-answer, the split is 27.1% versus 9.2%.Source: §5 · p. 7

Personalization case study

• On PersonaMem-v2, MEMORILLA uses remembered iron-deficiency anemia history to personalize a generic query despite no lexical overlap with the relevant conversation.Source: Figure 6 · p. 8

Reinforcement-learned memory

• In TextWorld, the figure contrasts ICL looping between explored rooms with MEMORILLA reaching the goal; its memory module is trained from sparse rewards while the encoder and LLM remain frozen.The setup compresses trajectories to K=8 latents and uses G=16 rollouts for GRPO updates to the memory module alone.Source: Figure 7 · p. 8

What neither method solves

• On PV4, many questions both methods miss require linking facts found somewhere in the user's history; on FactKG, 89.5% of claims both miss are true, versus 28.7% of claims both answer correctly.The passage suggests the PV4 errors involve combining information and the FactKG pattern indicates a miscalibrated decision threshold rather than missing evidence.Source: §5 · p. 8

6 CONTINUAL LEARNING WITH MEMORILLA

• MEMORILLA stores each completed turn as a new document and condenses the growing store into K = 8 memory tokens before the prompt.The agent explores up to 65 rooms and pursues multi-step goals for at most 50 turns.Source: §6 · p. 8

6.1 EXPERIMENTAL SETUP

• The frozen Qwen3-4B-Instruct-2507 decoder is paired with a frozen Qwen3-Embedding-4B encoder, and only the memory module is trained with GRPO.The memory supplements rather than replaces the full history; all conditions share the same input budget, so gains reflect learned extraction rather than extra information access.Source: §6.1 · p. 8

6.2 RESULTS

• Training MEMORILLA with GRPO raises avg@3 from 0.186 to 0.242 and pass@3 from 0.250 to 0.375 versus full-history ICL.Untrained memory falls below ICL on both metrics, indicating that the gain comes from learning rather than the added tokens.Source: §6.2 · p. 9, Figure 8 · p. 9
• Trajectory inspection shows trained memory helps the agent escape navigation loops and execute keyed action sequences, while ICL cycles among explored rooms.The remaining failures cluster in the longest games, whose quests approach or exceed the 50-turn limit and require extensive backtracking.Source: §6.2 · p. 9

7 DISCUSSION AND CONCLUSION

• The discussion frames MEMORILLA as separating persistent knowledge storage from per-query context, with learned memory tokens drawing on growing histories without proportionally expanding decoder context.The analyses suggest learned representations capture cross-document relationships and recover personal context beyond lexical similarity.Source: §7 · p. 9
• MEMORILLA leads retrieval, compression, and agent-memory baselines on average at 20 to 4,212× fewer document tokens across personalization, long-document QA, and knowledge-intensive tasks.The conclusion also reports continual learning in agentic RL by training only the memory module, without overwriting prior knowledge.Source: §7 · p. 9
• MEMORILLA’s limitations include possible loss of fine-grained facts, untested transfer across model families, and uncertain direct applicability beyond personalization, long-document QA, and knowledge-intensive QA.The authors suggest compression-trained encoders and combining latent memory with text retrieval as future directions.Source: §7 · p. 9
• For real-user deployment, stored embeddings may encode sensitive information, so the authors say deployments should obtain consent and protect them.PersonalizationV4 instead uses synthetic users based on LLM-generated personas and contains no real-user data.Source: ETHICS STATEMENT · p. 10

A RELATED WORK

• Textual memory preserves provenance and exact details but depends on selected records; latent compression is query-independent, while model memory couples recall to architecture-specific state.MEMORILLA instead reads an external document store into query-conditioned input embeddings for a frozen LLM.Source: A RELATED WORK · p. 15

B.1 TRAINING DATASETS

• The finetuning mixture contains 1.34M queries across 10 sources spanning reading comprehension, open-domain QA, and summarization.The pretraining corpus comprises approximately 2M paraphrase-reconstruction examples derived from English Wikipedia.Source: §B.1 · pp. 15–16, Table 1 · p. 16

B.2 TRAINING CONFIGURATION

• All runs freeze encoder and decoder backbones; only the memory module is trainable unless otherwise noted.Document embeddings are precomputed and cached, and supervised training uses bf16 mixed precision on four NVIDIA H100 80GB GPUs with seed 23.Source: §B.2 · p. 16
• A single memory module is trained jointly across downstream tasks for up to five epochs, with extra epochs for PV4 and PMv2 because validation performance was still improving.Train and test users are disjoint for both benchmarks, so enrolling a new user requires only embedding their chats.Source: §B.2 · p. 16
• Table 2 specifies document chunks per benchmark, with each chunk embedded once by the frozen encoder and token counts measured using the Qwen3 tokenizer.Source: Table 2 · p. 16

C ADDITIONAL DETAILS ON BASELINES

• All baselines use frozen Qwen3-8B decoding and frozen Qwen3-Embedding-4B encoding, with greedy answer generation.Text-inserting baselines share a benchmark-specific prompt template and differ only in the context placed before the question.Source: C ADDITIONAL DETAILS ON BASELINES · p. 16
• RAG-k places the top-k chunks ranked by query–chunk embedding cosine similarity into the prompt as text, while Full Context places every chunk in stored order.Inputs beyond the enforced 32,768-token limit use YaRN scaling to extend the window to 131,072 tokens; earliest chunks are dropped for 10.8% of evaluated NarrativeQA questions.Source: C ADDITIONAL DETAILS ON BASELINES · pp. 16–17
• LLMLingua-2 compresses documents by deleting tokens, using a question-independent classifier and absolute token budgets rather than a compression rate.Its prompt retains the question, answer-format instruction, system prompt, and chat template unchanged.Source: C ADDITIONAL DETAILS ON BASELINES · pp. 17–18
• xRAG maps the embedding of the top cosine-retrieved chunk to one decoder-space soft token through a trained bridge.The bridge is trained from scratch on each benchmark’s training split.Source: C ADDITIONAL DETAILS ON BASELINES · p. 18
• Mem0 consolidates memories at write time, while MemoRAG forms clue queries from global memory to guide retrieval; both use official implementations and swept retrieval depths.MemoRAG’s long-context mode reads the uncompressed collection, and its clue mechanism is the only difference from RAG-k.Source: C ADDITIONAL DETAILS ON BASELINES · p. 18

D ADDITIONAL DETAILS ON BENCHMARKS

• Personalization tasks test combining past conversations or writing, long-document QA tests representing full sources for QA, and knowledge-intensive QA tests factual grounding from structured or semi-structured evidence.The benchmark overview also reports standard splits and disjoint training and evaluation users for personalization.Source: D ADDITIONAL DETAILS ON BENCHMARKS · pp. 18–19

D.1 PERSONALIZATIONV4 DATA PIPELINE

• PersonalizationV4 derives 200 two-turn conversations per user from synthetic personas, then poses free-text questions requiring multiple persona facts; its 4,237 held-out questions come from 30 users.The 629-question hard subset contains items the untrained decoder answers incorrectly without documents, with top-1 retrieval, or both.Source: §D.1 · p. 20

D.2 CONTEXT BUDGET PER QUERY

• MEMORILLA contributes 16 document-derived tokens per query, while xRAG contributes one; the context-budget table reports mean document-token counts and whether full-context prompts fit 32K.The compression column compares Full Context tokens with MEMORILLA’s 16 tokens.Source: §D.2 · p. 21, Table 10 · p. 22

E ADDITIONAL RESULTS AND ABLATIONS

• Table 11 presents the complete eight-benchmark results and reports compression as the Full Context document-token count divided by MEMORILLA’s 16 tokens.Its caption specifies frozen Qwen3-4B embeddings and Qwen3-8B decoding; MEMORILLA uses K=16.Source: Table 11 · p. 22, E ADDITIONAL RESULTS AND ABLATIONS · p. 22

E.1 MEMORY ARCHITECTURE

• 82.56 versus 80.48 on FactKG, MEMORILLA outperforms the Q-Former, while query-independent pooling is weakest and learned queries conditioned before document reading account for the gains.Replacing document embeddings with Gaussian noise lowers MEMORILLA by 5.93 and the Q-Former by 5.11, but changes the documents-only result by only +0.04.Source: §E.1 · p. 23

E.2 MEMORY CAPACITY AT TRAINING TIME

• Four of eight benchmarks change by less than a point from K=16 to K=128, while PMv2 and PV4 decline, supporting K=16 in the trained setting.The comparison trains both capacities from scratch under otherwise identical settings.Source: §E.2 · p. 23, Figure 9 · p. 24

E.3 MEMORY TOKEN BUDGETS FOR INFERENCE

• At inference, personalization gains most from later slots, while FactKG, TriviaQA, and NarrativeQA improve with each doubling; PubMedQA and LaMP-4 improve from K=8, and LaMP-7 from K=16.The tested tokens are dropped from a K=16-trained checkpoint without retraining; the paper relates capacity needs to cross-document integration demands.Source: §E.3 · pp. 23–24, Figure 10 · p. 25

E.4 FROZEN VS. LORA TRAINING

• MEMORILLA leads the comparable-budget no-document LoRA adapters on every primary metric, though the PubMedQA and LaMP-7 gaps are within noise.The comparison uses an 8B decoder with 194.1M memory-module parameters against a 4B decoder with 268.4M LoRA parameters.Source: §E.4 · p. 24

E.5 ROLE OF ENCODER SIZE

• Every score changes by at most 1.3 when the frozen encoder grows from Qwen3-Embedding-4B to 8B, except PubMedQA, which gains 3.00.That PubMedQA change corresponds to three of 100 questions.Source: §E.5 · p. 24

F COMPUTATIONAL ANALYSIS OF MEMORILLA

• With a 512× larger collection, MEMORILLA end-to-end latency rises only 1.9 ms, while full-context decoding grows by 19,184 ms and exceeds the 32,768-token limit.The measurements use one H100-80GB at batch size 1; the full-context case beyond the limit uses YaRN.Source: F COMPUTATIONAL ANALYSIS OF MEMORILLA · p. 25, Table 14 · p. 26
• MEMORILLA’s document encoder runs once on the write path, and its stored embeddings are reused across queries; its read path supplies the decoder a fixed K=16 tokens.Dense retrieval shares the one-time encoding cost, while RAG’s decoder prefill grows with retrieval depth.Source: F COMPUTATIONAL ANALYSIS OF MEMORILLA · p. 25
• On PV4, the document store is 7,266× smaller than the corresponding decoder KV cache, while transfer remains under 0.04% of query latency.The analysis says the benefit is serving cost that stays flat as collections grow and is amortized across queries, not faster latency for one query.Source: F COMPUTATIONAL ANALYSIS OF MEMORILLA · p. 25

G CONTINUAL LEARNING WITH MEMORILLA: ADDITIONAL DETAILS

• From the second turn, completed TextWorld turns become documents; the current observation queries them, and eight memory vectors precede the prompt while the full trajectory remains present.For 20 extreme, huge, long, and mega test games, one document represents each five completed turns, with full observations and the mean document embedding as query.Source: G CONTINUAL LEARNING WITH MEMORILLA:… · p. 26, Figure 13 · p. 27
• Training uses 800 prompts and held-out validation and test splits of 40 games each, with three sampled completions per held-out game; the random-memory baseline uses untrained memory weights.Games span six tiers, with 15–65 rooms and 6–74-step quests.Source: G CONTINUAL LEARNING WITH MEMORILLA:… · p. 26, Table 15 · p. 28

G.1 REINFORCEMENT-LEARNED MEMORY OBJECTIVE

• GRPO trains only the memory module from environment rewards, standardizing each rollout’s scalar reward against the other rollouts in its group before the clipped PPO-style update.The update is applied over generated action tokens.Source: §G.1 · p. 27
• Turn rewards are unshaped TextWorld scores summed across the rollout; the memory supplements rather than replaces the trajectory and is recomputed from completed turns using the current observation as query.Longer games use a mean document embedding as the query.Source: §G.1 · pp. 27–28