Source-linked AI summary
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
Asa Shepard
TL;DR
The paper asks when precomputed memories remain reliable as material is composed, updated, or corrected. It evaluates caches and trained cartridges on Llama-3.1-8B-Instruct, finding composition penalties, replay-dependent rebuilding, and weak query-time correction that improves sharply when updates name their questions.
Problem
The paper studies whether precomputed memories can be composed, rebuilt cost-effectively, and corrected as information changes, rather than only serving static corpora.
Method
The paper compares saved key-value caches and trained cartridges across composition, warm-rebuilding with replay, and query-time update experiments against full-context references.
Results
Precomputed memories degrade when separately prepared parts are composed; replay preserves warm-rebuild quality, while query-time corrections work much better when they name the question.
Takeaways & Limitations
Precomputed memories should be rebuilt as new information changes their source material, with warm replay and specifically phrased updates offering supported ways to maintain currency.
Takeaways & Limitations
The conclusions are bounded chiefly by one model, tested lengths up to 24,000 tokens, 512 updates, and one warm-start rule, corpus, cartridge size, and seed.
Abstract
from arXiv · showhide
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
1 Introduction
The paper tests whether precomputed memories can be composed, rebuilt, and corrected as information changes. It finds composition penalties, replay-dependent warm rebuilding, and phrasing-sensitive query-time corrections.
- Scope: Precomputed memories include one-pass key-value caches and smaller trained cartridges reused across diverse queries about a common corpus.Caches are cheap and text-sized; cartridges are costly to train but compressed for amortization across future queries.
- Preparation: Preparation is treated as requiring full joint exposure because cached token states depend on attention to all other tokens in the same pass.A change to earlier material can therefore alter later stored states, and cartridges inherit this dependency through self-study.
- Composition: Separately trained cartridges lose accuracy when composed, with about half the observed loss attributed to duplicated start-of-text markers.The remaining loss persists after partially co-visible training.
- Rebuilding: Warm rebuilding remains within ten points of fresh retraining only when older training data is replayed during successive updates.No replay is 15× cheaper per rebuild but produces no measurable benefit across thirteen updates; half replay matches full replay at about half fresh-retrain cost.
- Correction: Query-time corrections are used on about a third of two-hop questions, fall below a tenth after 512 revisions, and leave every measured policy at least 20 points below full-context reference.An update naming its question is used on over nine-tenths of items, while a value naming nothing is used on a tenth; text versus cache injection differs by less than a point.
2 Related work
Related work frames cartridges as compressed prefix-like memories, query-time correction as knowledge conflict, and retrieval as an apparent response to staleness. The paper finds that even effectively gold retrieval remains substantially below full-context performance on two-hop questions.
- Compressed memories: Cartridges distill text into small trained key-value caches through self-study, which reproduces behavior as if the full text were in context.Plain next-token prediction performs far worse than placing the material in context, making self-study necessary.
- Knowledge conflict: The measured query-time failure is a knowledge conflict: stale text can override a correction presented in the same context.This connects the result to entity-based conflicts, contradiction faithfulness, and parameter-level editing failures on multi-hop questions.
- Retrieval: Retrieval is tested with BM25 and pseudo-relevance feedback, whose expanded query ranks the correct update first at every tested pile size.This makes the retrieval arm effectively gold retrieval and the strongest policy tested at large piles.
- Retrieval: Even effectively gold retrieval ends at least 64 points below the full-context reference on two-hop items.The gap may also reflect the model’s baseline reasoning capability.
3 Reading the results
The results use deterministic experiments centered on Llama-3.1-8B-Instruct, with caches and cartridges evaluated on synthetic and clinical-record tasks. Performance is compared with full-context reading, no relevant text, and correction-specific rescue-rate scoring.
- Experimental scope: Except where noted, all results come from deterministic bf16 Llama-3.1-8B-Instruct experiments, with fresh-item replication rather than reruns.A descriptive second-model check appears separately, so the main verdicts concern this model.
- Substrates: The study evaluates prefilled caches and trained cartridges as two distinct precomputed-memory substrates.The head-to-head comparison is collected in Section 7.
- Texts and questions: Synthetic texts span roughly 1,500–24,000 tokens, while cartridge experiments use patient records from the LongHealth benchmark.Questions are single-hop or two-hop, with the latter requiring two separately stated facts to be combined.
- Scoring: Full-context reading is the upper reference, no relevant text is the floor, and correction experiments report rescue rate rather than ordinary accuracy.Rescue rate is the fraction of guaranteed failures a policy repairs, with correction floors and ceilings fixed by construction.
4 Build time: composing separately built memories
Separately trained cartridge memories can be composed on demand, but composition substantially reduces accuracy. Removing duplicated BOS columns recovers about half the loss, while the remaining degradation persists under the tested co-visible training setup.
- Analysis: Duplicated BOS columns account for about half of the composition loss, whereas deleting an ordinary column changes nothing.Each separately built cartridge contains its own BOS column, so an 8-cartridge composition has eight BOS columns instead of one.
- Results: Composition reduces accuracy as the number of separately trained cartridges increases, relative to an item-matched single-cartridge baseline.The evaluation compares compositions of 2, 4, and 8 cartridges against a single-cartridge baseline scored on each item’s source record.
- Results: At 8 memories, composition causes a 26.25-point collapse, while removing extra BOS columns recovers 13.75 points and leaves a 12.50-point net penalty.The recovery is statistically significant (p = 4.7 × 10^-4) and exceeds the predefined half-collapse threshold by 0.625 points.
- Analysis: Thirty-percent co-visible training does not protect composition: at 8 memories, both few-source and many-source arms incur a −22.5-point penalty.Both arms end near the no-memory floor, so the data cannot determine whether the number of co-visible sources matters.
- Analysis: Removing every BOS column destroys the memory, reducing accuracy to 0.106 and selecting the same option on all 160 items.The result indicates that one start marker is required, while additional markers are harmful.
- Implications: The surviving half of the loss remains after the tested repair, although other positional, token-handling, calibration, recomputation, or co-visibility strategies were not tested.The practical intervention supported here is stripping duplicated start-of-text columns down to one when composition is unavoidable.
5 Build time: retraining as sources accumulate
Warm-start rebuilding can maintain a growing cartridge memory, but preserving quality requires replaying older material; half replay here nearly matches full replay at roughly half the cost of fresh retraining.
- Setup: The chain adds one record per generation for 13 generations, warm-starting each cartridge from its predecessor at replay fractions 0, 0.5, and 1.Every cartridge uses a 2,048-token budget, and replay mixes earlier sources’ self-study examples with the new source’s data.
- Quality: Replayed chains remain within the corpus’s 10-point detection threshold of fresh retraining through generation 13, so their tolerable chain depth is more than 13.This lower-bound result applies to replay fractions 0.5 and 1.
- Quality: At generation 13, full replay reaches 0.5250 versus 0.3750 for no replay, a 15.00-point gap, while the chains are indistinguishable through generation 4.The no-replay chain also falls 8.93 points below fresh retraining and 11.79 points below the initialization-matched retrain.
- Quality: Thirteen no-replay updates add only +2.50 points over the stale floor, compared with +16.79 for half replay and +17.50 for full replay.The no-replay gain is not measurable over never updating, whereas replay preserves accumulated information.
- Cost: At generation 13, half-replay rebuilding costs about half a fresh retrain and reaches 0.5179 versus 0.5250 for full replay, a difference the corpus cannot distinguish.No-replay rebuilding is roughly 15× cheaper but is the arm that decays; full replay costs about 6% more than fresh retraining.
- Scope: The experiment prices two accumulation paths on one corpus: one-memory-per-source composition and one evolving memory maintained through replay.A bounded working set of mutually conditioned cache chunks is only a precursor measurement.
6 Query time: serving corrections beside a built memory
Serving corrections beside a built memory helps only briefly and unreliably: a lone correction rescues about a third of two-hop failures, while accumulating revisions sharply reduce use. Wording that names the question improves binding, but text delivery generally outperforms injection as competing revisions accumulate.
- Single corrections: About 0.34 of items are rescued by a single relevant correction, whether delivered as pasted text or conditioned injection.Only independent injection performs differently, rescuing 0.031 of items.
- Accumulating revisions: At pile size 512, tuned retrieval retains 0.235 rescue rate, versus 0.090 for pasting everything and 0.016–0.067 for injection policies.Pasting merged reaches 0.220 and paste by relevance 0.196 at the same pile size.
- Delivery mechanisms: The best text policy beats the best injection policy at 8 of 10 pile sizes, with a pooled lead of 9.45 points.Pooled rescue is 0.287 for tuned retrieval and 0.192 for merged injection; the methods are equal at pile size 1.
- Question tiers: Single-hop rescue rises 16.7 points from pile size 1 to 32 and then plateaus, whereas two-hop rescue never gains and ends 10.6 points below pile size 1.Each tier uses its own best text policy and baseline, so the curves should not be read as a direct tier comparison.
- What makes updates bind: Naming the question is the decisive update feature: the factorial contrast is +83.3 points, while delivery mechanism changes no cell by more than a point.A false value with the question named is adopted on 0.877 of items, indicating compliance rather than recognition.
- Interpretation: Corrections served beside stale memory still fail on two-thirds of items, even when pasted text rereads the stale text and correction together in one fresh prefill.This places the single-update barrier at the level of knowledge conflict rather than delivery machinery.
7 Discussion
Across substrates, precomputed memory’s main constraints arise from the workload and update policy rather than the substrate alone. The results favor joint preparation, replayed warm rebuilds, and carefully named query-time updates, with routing by query complexity as a practical design.
- Cross-substrate comparison: Cache composition loses 48.5 to 79.5 points, while cartridge composition loses 10.6 to 12.5 net points, because the questions differ in cross-source linking demands.The piece deficit affects assembled caches, whereas the smaller cartridge penalty reflects workloads whose clinical questions do not require links across sources.
- Workload split: Lookup-style workloads tolerate partial preparation, staleness, and accumulated updates better than synthesis-style workloads requiring multi-hop reasoning.Routing multi-hop questions to full-context reads while serving single-hop questions from memory concentrates memory use where it survives best.
- Rebuilding: A full-replay warm rebuild costs slightly more than fresh retraining, while half replay matches it within the detectable margin at about half the cost.No-replay rebuilding is 15× cheaper per rebuild but produced thirteen successive updates with no measurable benefit over never updating.
- Open questions: The paper leaves several measurements open, including longer contexts, larger compositions, intermediate replay fractions, and whether named updates remain bound after accumulating revisions.These are presented as informed speculation rather than measured results.
- Query-time updates: Naming the question an update answers raises binding across caches and cartridges, but falsely named updates are adopted nearly as readily as true ones.Pasted text and cache injection perform similarly, while the wrong-naming control indicates compliance rather than simple recognition.
8 Limitations and scope
The evidence is bounded by one primary model, finite text and update ranges, and specific experimental configurations. The paper also identifies a practical reliability risk: stale memories can repeat outdated facts, while named corrections can propagate false values.
- Model scope: All primary quantities come from Llama-3.1-8B-Instruct, so the paper cannot claim behavior for language models generally.A Qwen3-8B check repeats the direction but is an estimate, not a systematic model comparison.
- Length scope: The tested lengths stop at 24,000 tokens, and the measured slope constrains nothing beyond that range.Longer contexts could change both the constant-fraction pattern and the full-context reference.
- Experimental boundaries: Most quantities are bounded by tested ranges including 13 generations, 512 updates, revision depth 5, and pile size 1.The warm-start result also depends on one rule, corpus, cartridge size, and seed.
- Reference and economics: The full-context reference uses corrected text alone, so the study does not establish the gap to a reference carrying the same update pile.It also does not establish a numeric amortization break-even without a workload query rate.
- Deployment boundary: Serving stale precomputed memory without rebuild discipline repeated outdated facts on most tested items, creating a reliability risk for clinical or legal assistance.Question-named updates are a large replicated lever, but false named values are adopted nearly as readily as true ones.
A.5 Details relocated from the experimental sections
The relocated details clarify the statistical limits and measurement design behind composition, retraining, and query-time update results. They show where comparisons were resolvable, where effects remained within detectable margins, and how update wording was tested.
- Composition: At two memories, composition penalties were −0.6 and −5.0, neither significant; at four and eight memories, both arms reached −12.5 and −22.5, respectively.Because the registered precondition failed, the paper draws no conclusion about source-count effects.
- Warm retraining: A generation-8 full-against-no-replay difference of +7.50 points was significant at p = 0.026 but remained inside its 8.97-point minimum detectable effect.The study recorded several comparisons as unresolved rather than estimating beyond measurable power.
- Staleness screening: Among 328 screened items, 198 produced the outdated value under stale reading after distractor failures were removed.The text-over-injection comparison favored text by 9.45 pooled points, with MDE 2.26 and p = 4.0 × 10−33.
- Update wording: The registered naming contrast was +83.3 points on 228 items and +82.3 on the 198-item population.The clinical anchor addendum improved superseded-fact accuracy from 0.105 to 0.655, replicating at +45.0 points on held-out patients.
B Build time: how much co-preparation a memory needs
Partial preparation preserves performance on cross-paragraph reasoning only when a large fraction of the text is jointly visible. Separate pieces omit the relationships needed for multi-hop answers, and this requirement remains constant across the tested length range.
- Threshold scaling: Two-hop threshold τ remains 0.82 to 0.92 at every tested length, with slope −0.045 and confidence interval −0.130 to +0.046.The measured slope lies within the preregistered constant-fraction band and far from the fixed-token-count band.
- Two-hop composition: At f = 0, isolated pieces score 0.195 to 0.510 on two-hop questions versus 0.985 to 1.000 after joint prefill, a deficit of 48.5 to 79.5 points.The experiment assembles separately prefilled, paragraph-bounded pieces while retaining all text content.
- Single-hop contrast: Single-hop accuracy at f = 0 reaches 0.635 to 0.795 in eight-piece configurations because the relevant fact remains intact within one piece.Single-hop thresholds are 0.34 to 0.58, below the two-hop range of 0.82 to 0.92.
- Economic implication: Reaching 95% of fully prepared two-hop accuracy at 24,000 tokens requires roughly 20,000 co-prepared tokens.Thus partial preparation does not make preparation cheap over the tested range; reuse savings come from paying full preparation once and querying repeatedly.
- Scope: The tested lengths span 1,500 to 24,000 tokens, but the slope constrains nothing beyond 24,000 tokens.At longer lengths, long-context degradation could also reduce the full-prefill reference.
C Query time: repeated revisions to one fact
The revision experiment tests whether query-time updates can keep a static memory current as one fact changes repeatedly. Revisions create a small ordering advantage, but merged injection does not overcome the broader correction problem.
- C Query time: repeated revisions to one fact: The experiment targets histories in which one fact has multiple versions but only the latest version remains current.Relevance ordering cannot encode which version is newest when all versions are equally relevant to the question.
- C Query time: repeated revisions to one fact: The experiment crosses pile sizes 32 and 256 with revision depths 1, 2, 3, and 5 and with 1, 4, or 16 simultaneously stale facts.Depth counts supersessions of one fact, and accuracy is reported as rescue rate on forced-choice items.
- C Query time: repeated revisions to one fact: At revision depth 5, newest-first rescues 0.197 versus 0.155 for relevance ordering, a paired difference of +4.24 points.The difference is significant (p = 3.3×10−6) and exceeds the MDE of 2.53.
- C Query time: repeated revisions to one fact: The additional decay from revision ordering is about 1.6 points per revision, one-third of the size the detection rule targeted.This is the difference between the two policies’ least-squares decay slopes across depths 1, 2, 3, and 5.
D.1 Evaluator choices
Evaluator choices materially affect measured accuracy in these experiments. Answer format produces the largest shift, while an incorrect chat template also causes substantial losses.
- D.1 Evaluator choices: Answer format moves accuracy by 64 points on this instrument.The evaluation varied multiple-choice and free-form answers.
- D.1 Evaluator choices: The wrong chat template costs 4.76 points under multiple choice and 5.81 under free-form.These values are pooled over 20 cells at n = 3,991.
- D.1 Evaluator choices: On two-hop questions, the bad template costs 8.45 to 10.25 points.The passage reports this range separately from the pooled all-cell losses.
D.2 The honest ceiling
The paper measures memory performance against full-context reading on LongHealth using fictional patient records and an evaluation designed to avoid strategy-dependent item selection. This establishes the benchmark setting for an honest comparison.
- D.2 The honest ceiling: The LongHealth evaluation contains 400 five-option questions over records from 20 fictional patients.Each record contains 5,090 to 6,754 words, or 9,500 to 12,400 tokens under this paper’s tokenizer.
- D.2 The honest ceiling: The study measures the best trained cartridge across a budget sweep against the usual full-context reference.It also builds a second evaluation whose items are selected independently of any strategy’s performance.
D.3 Distractor confusability has two effective levels
The distractor evaluation finds two effective difficulty levels rather than the intended three-rung ladder. Mixed-type distractors are substantially easier, with the confusability effect depending on answer format and question type.
- D.3 Distractor confusability has two effective levels: The evaluation crosses near-identical, same-type, and mixed-type regimes with two question tiers across five text lengths, yielding 47,928 measurements.There are 200 items per cell, and lengths are pooled by design to prevent post-hoc selection of a favorable length.
- D.3 Distractor confusability has two effective levels: Near-identical and same-type distractors are indistinguishable, at +3.83 and +5.52 points with neither significant.The intended three-rung ladder therefore has two effective levels.
- D.3 Distractor confusability has two effective levels: Mixed-type distractors are 18 to 47 points easier than the other two regimes.The passage states that all the difficulty comes from mixed-type being easier.
- D.3 Distractor confusability has two effective levels: The confusability penalty is 14.13 points larger for two-hop questions under multiple choice, exceeding the prespecified threshold of 10.Under free-form answers, the same contrast is −5.01 and misses the threshold.
- D.3 Distractor confusability has two effective levels: A precursor study on Llama-3.1-8B-Instruct found modularizing a co-loaded cache into five conditioned chunks cost no accuracy, but it used a different benchmark and judge.That precursor is reported for grounding only and carries no evidential weight in this paper’s claims.