Source-linked AI summary
Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
Ankit Goyal, Jaideep Ray
TL;DR
The paper asks whether agent memory remains usable when models, embedding indexes, or repair resources change. It compares four memory formats using controlled synthetic histories, randomized answer codes, exact scoring, and two open-weight models. Fixed-schema memory transfers reliably, while free-form notes and partially migrated indexes show direction- and retrieval-dependent losses, motivating migration testing and source-history retention.
Problem
Model upgrades can silently reduce memory performance because readers may reinterpret notes, embedding spaces may be incompatible, and repair may lack original evidence.
Method
The study compares four memory formats under controlled migrations, isolating writer–reader pairs, embedding models, repair sources, and diagnostic evidence.
Results
Fixed-schema structures transfer reliably, whereas NOTES show asymmetric model coupling and mixed embedding indexes recover only part of the gain from full re-embedding.
Takeaways & Limitations
Treat model upgrades as memory migrations, test directions separately, isolate embedding spaces, and retain source histories when policy allows.
Takeaways & Limitations
The findings come from a single-stage dense retriever, two similarly sized models, and scripted objective workloads whose histories fit within both context windows.
Abstract
from arXiv · showhide
Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ($0.467 \pm 0.014$) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ($0.364 \pm 0.012$) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.
1 Introduction
Agent memory persists experience outside the model, but upgrades can silently reduce performance when writers, readers, or retrieval components change. This study compares four formats under controlled migration conditions to identify portability risks and recovery requirements.
- Memory formats: Four formats preserve memory differently: LC-RAW keeps full history, RAG retrieves transcript chunks, NOTES stores compressed summaries, and KG-fixed stores schema-constrained claims.These formats trade off context use, retrieval dependence, compression loss, and schema coverage.
- Upgrade risk: Memory can fail silently after upgrades when new readers interpret notes differently, embedding spaces become incompatible, or repair lacks the original evidence.The database may continue returning results even as performance drops.
- Research gap: The study asks how common memory formats compare under identical migration conditions, extending prior work on cross-model transfer, graph memory, procedural memory, and production migration tools.The authors identify a remaining engineering question about format dependence under controlled conditions.
- Study design: The framework separates memory writing from reading and evaluates writer–reader pairs, embedding models, and repair sources one variable at a time.Randomized answer codes and exact scoring reduce reliance on pretraining or model-based judging.
- Contributions: The paper contributes controlled comparisons, direction-specific migration measurements, tests of mixed indexes and repair sources, and a loss breakdown across writing, retrieval, and reading.It also presents a practical design playbook for memory migration.
2 What does it mean for memory to survive an upgrade?
The study defines memory survival as performance retained when a changed component uses an existing store, then measures portability and recovery across writing, indexing, reading, and repair. RPAS captures performance after a writer–reader swap, while CTR captures the cheapest repair reaching a target recovery level.
- System model: An agent memory system includes an actor, writer, reader, and embedder, any of which may change during an upgrade.The framework holds the history fixed while varying the component responsible for writing, indexing, reading, or repair.
- RAG pipeline: RAG chunks histories, embeds chunks and queries with the same configuration, ranks vectors by similarity, and sends top-ranked text to the reader.Changing the embedder requires full re-embedding because old and new vectors occupy different spaces even when dimensions match.
- Memory formats: LC-RAW keeps complete transcripts, RAG retrieves chunks, NOTES compresses histories into summaries, and KG-fixed stores schema-constrained claims.Figure 1 compares each format’s persisted state, read path, and primary upgrade risk.
- Migration metrics: RPAS measures performance retained when a reader inherits another model’s store relative to its own-store performance.An RPAS of 1 indicates no performance loss, while 0.8 indicates 80% retention of own-store performance.
- Migration metrics: CTR is the cheapest repair method and budget that restores at least a chosen fraction of the new reader’s own-store performance.Raw-retained repair can recover omitted or distorted facts, whereas store-only repair cannot recreate absent evidence.
- Evaluation: The study overview varies one component at a time and reports exact-match accuracy, RPAS, and CTR across model, embedding-index, repair-source, and diagnostic changes.Together, these measures connect upgrade survival with recovery cost.
3 Testing memory migrations
The study tests memory migrations with controlled histories, exact scoring, and predefined contrasts across model, embedding, and repair changes. It uses validation checks, signed analysis plans, and per-history comparisons to isolate migration effects.
- Evaluation design: 48 synthetic histories use randomized codes and exact scoring to identify where memory facts are lost.Each history contains questions spanning direct facts, temporal changes, contradictions, multi-step relations, and aliases.
- Evaluation design: Two fixed open-weight models, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M, test one cross-family migration.The design does not sample all possible model upgrades.
- Experimental controls: The study matches token, storage, context, retrieval, and repair budgets where formats permit direct comparison.Storage is reported in KiB, NOTES are capped at 128 KiB, and readers use the same context budget and RAG settings.
- Analysis plan: The four planned tests use one-sided tests against a five-point threshold with Holm correction across four hypotheses.The later bootstrap used 10,000 resamples and reproduced the same decisions, but it was run after the t-test results were observed.
- Experimental controls: The evaluation covers writer migration, partial embedding migration, and controlled evidence-repair tests across four migration tests.The writer-migration test includes both directions and 576 evaluations across 48 histories; the mixed index represents silent partial migration.
4 Results and analysis
The results show that memory portability depends on representation and migration path: fixed-schema records remain stable, while notes and retrieval systems lose performance through different mechanisms. Repair is also conditional on retaining raw history and on the repair model’s resource limits.
- Fixed-schema portability: KG-fixed changes by only +0.0004 ± 0.0020 after a writer swap, while its own-store accuracy remains 0.845–0.988.The shared schema fixes keys, relation names, source fields, and reading rules; models fill only the values.
- Fixed-schema portability: NOTES transfer is strongly direction-dependent: Llama-written notes reduce Qwen accuracy by 13.3 points, whereas Qwen-written notes improve Llama by 9.9 points.The asymmetry reflects underlying model capabilities rather than a restrictive prompt or byte limit.
- Retrieval migration: The tested RAG pipeline reaches only 0.535 to 0.565 accuracy versus 0.712 to 0.911 for LC-RAW, with retrieval contributing 81% of RAG’s deficit.The retrieval design used a single-stage dense retriever with event-based chunks, cosine top-k=8, and no reranker or lexical search.
- Retrieval migration: Full re-embedding gains 11.90 points over the old index, but a 50/50 mixed index gains only 4.96 points because same-dimensional vector spaces remain incompatible.The system continues running without dimensional compatibility errors, while recall and ranking degrade.
- Repair: Store-only NOTES rewriting never reaches 90% of own-store performance, while raw-history repair reaches the threshold for 34 of 48 histories in one direction.Qwen reconstructs 34 histories at roughly $0.76 median cost, whereas Llama reconstructs none because attempts hit the output-token limit.
- Diagnostic decomposition: NOTES loses 0.467 ± 0.014 of its 0.584 ± 0.013 pooled mean deficit during writing, whereas RAG loses 0.364 ± 0.012 of its 0.450 ± 0.012 deficit during retrieval.These correspond to descriptive shares of roughly 80% and 81%, respectively.
5 Agent Memory Compatibility Across Model Transitions
Memory compatibility depends on how information is represented, built, retrieved, and read across model transitions. The study emphasizes explicit compatibility checks, provenance, retained evidence, and stage-specific diagnosis.
- Memory formats trade off fidelity, compactness, and compatibility, with compressed representations potentially losing information that migration cannot reconstruct.
- Recording the model, prompt, schema, embedding, chunking, and index configuration improves failure attribution.
- Compatibility is direction-specific: averaging transitions can conceal substantial failures in one migration direction.
- Embedding changes can alter retrieved evidence independently of reader quality, so final-answer evaluation can miss migration regressions.
- Migration errors can arise from retention, retrieval, or reading, which are experimentally distinct failure stages.
6 Conclusion
The conclusion treats upgrades as memory migrations rather than simple model replacements. It recommends testing inherited stores, rebuilding indexes, retaining protected source histories when permitted, and diagnosing the failed stage before repair.
- Treat a model upgrade as a memory migration, not as a simple replacement.
- Migration testing should identify whether information was lost during writing, retrieval, or reading before repair is chosen.
- A durable memory system should let future models read, verify, and rebuild the store when necessary.
A What this study adds to prior memory work
The study addresses a measurement gap in prior memory work by comparing common memory designs under the same model change, retrieval settings, and repair budget. Its practical contribution is controlled measurement of accuracy loss and repairability.
- Prior work reports transfer failures or robustness under particular designs, but this study compares formats under a common migration setup.
- The study compares common memory designs under the same migration conditions rather than reproducing exact deployed systems.
- The practical gap is controlled measurement of how much accuracy memory migration loses and whether the store can be repaired.
- Evaluation should hold model change, retrieval settings, and repair budget constant because day-one accuracy does not establish upgrade survival.
B Why 48 histories are enough for this stress test
The study validates its evaluation before using it for sample-size planning, then uses evidence coverage and variance checks to justify a 48-history stress test. The design remains bounded by the tested writer behavior and evidence-preservation gate.
- Evaluation checks required memory removal to reduce accuracy, correct evidence to restore it, corruption to increase damage, and probe ranges to span at least 0.50.
- Initial planning used 12 histories to estimate variance and targeted 80% power for a five-point effect at a one-sided 5% error rate.
- No tested NOTES policy reached 90% evidence coverage, so the study lowered its minimum gate to 60% and recorded the change.
- In a 160 KiB sweep, Qwen retained 85.6% of required spans in approximately 143 KiB, while Llama retained 64.5% in approximately 159 KiB.
- Equal byte budgets do not guarantee equally informative stores, so evidence preservation must be measured alongside evaluation validation.
C How to use the 20-probe migration check
The 20-probe check can screen migration candidates, but its format-dependent bias and error make full evaluation necessary for approval.
- Predictive value: ρ = 0.860 measures how reasonably the 20-probe check tracks full accuracy across held-out predictions.Table 5 covers the other 140 probes across 576 held-out predictions and transfers across histories and model pairs.
- Bias and error: 0.031 is the check’s average performance overestimate across memory formats.Its slope of about 0.85 also pulls very high and very low scores toward the middle.
- Bias and error: MAE ≈ 0.10 is too large to resolve small migration effects such as the 0.0004 KG-fixed gap.The check remains nearly unbiased on unseen histories and model pairs, but its error is too large for final approval.
- Recommended use: Use the 20-probe check to rank candidates or stop an obviously bad rollout, then calibrate by format and run the full evaluation.The recommended workflow treats the probe check as an early-warning tool rather than an approval criterion.
D How to diagnose where memory fails
The diagnostic framework separates memory loss during construction, retrieval, and reading by comparing normal reading with supplied stored evidence and raw evidence. It identifies construction as the dominant NOTES deficit and retrieval as the dominant RAG deficit.
- Diagnostic design: The stored-evidence condition supplies exact answer-containing items, whereas the raw-evidence condition supplies the original event.Accuracy differences across these controlled conditions locate loss across construction, retrieval, and reading.
- Aggregation: The three component means add to total loss, while the reported percentages are descriptive ratios of pooled means.The differences are computed within matched cells before averaging the 48 history-level values, with 95% t-interval half-widths for absolute components.
- NOTES: 0.467 ± 0.014 of the 0.584 ± 0.013 NOTES deficit comes from construction, an 80% descriptive share.Retrieval within notes contributes 0.036 ± 0.009, or 6%.
- RAG: 81% of the RAG deficit is retrieval loss: 0.364 ± 0.012 of the 0.450 ± 0.012 pooled mean deficit.Supplying correct stored chunks raises accuracy from 0.53–0.56 to 0.88–0.95, while construction loss is 0.005 ± 0.005.
- NOTES: −0.012 ± 0.008 and −0.042 ± 0.017 show that rewriting NOTES in the new model’s style does not improve accuracy.The second intervention reached only 42% of intended notes, and Llama dropped exact identifiers in 46% of rewrites.
- Diagnosis to intervention: Measure NOTES evidence coverage and RAG recall before changing the reader, while treating these tests as localization rather than complete causal proof.The recommended interventions target writing for NOTES and chunking, indexing, or ranking for RAG.
E Which repairs work, and what they cost
Mechanical reconstruction is the most reliable repair strategy in these tests, whereas store-only NOTES rewriting cannot recover omitted evidence. Raw-history repair is direction- and model-dependent.
- Mechanical repairs: All 48 histories reach every target after full RAG re-embedding in both directions, at a median cost of $0.013.Rebuilding KG-fixed records into the shared schema is nearly universal and costs approximately zero.
- NOTES repair: 0/48 histories reach any recovery target when NOTES are rewritten from the store alone in either direction.A second rewrite cannot restore evidence omitted by the original notes.
- Raw-history repair: 34/48 histories reach 90% recovery with Qwen repairing from raw history, compared with 28/48 at 95% and 22/48 at 99%.The conditional median costs are $0.76, $0.75, and $0.75 for the progressively smaller successful subsets.
- Raw-history repair: 0/48 histories reach any target with Llama as the repair model, making Cost-to-Recover infinite across the tested grid.Raw history helps only when the repair model can use it.
- Design guidance: Prefer repairs that rebuild embeddings or canonical fields, and retain protected source history when NOTES may require repair.The exact repair model should be tested in advance because raw history alone does not guarantee recovery.
F How we made the runs auditable
The study makes migration runs auditable by preserving exact configurations and raw observations, enforcing live-service and embedding checks, isolating repeats, and recording every correction without overwriting originals.
- Replayability: Every run records code, settings, model and tokenizer versions, random seeds, and memory-store checksums.These controls support replay and traceability across the study’s evaluations and resource use.
- Validation: A run stops before invalid data when served model, tokenizer, context limit, embedding model, and vector size checks fail.For E3, the two embedding spaces had cross-space cosine 0.904, below the 0.999 identity threshold.
- Repeat isolation: 384,000–520,000 cache entries were scanned after repeat isolation, with no additional damaged entries found.Separate cache keys prevent answers from being reused across intended noise repeats.
- Resource controls: All 1,440 E6 evaluations completed in 98.2 GPU-hours despite two service outages.Run-control bugs were corrected by treating output-limit stops as failed repairs, charging reserved output, and preserving degraded stores as measured results.
- Change control: Original run data remain preserved when corrected indexes or analyses are created, with deviations, reasons, and affected-artifact checksums recorded.The fixed plan and statistical code are also stored for verification and reproduction.
- Audit standard: A trustworthy migration test should save exact versions, verify live services, stop on invalid configurations, isolate repeats, preserve raw observations, and log corrections.These controls define the study’s operational audit standard.