When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning

1. Introduction

• The motivating question is when data held in a context-computed state should instead become part of the model’s weights.Prior accounts describe stylized language and linear-regression models but provide no quantity for deciding between the two.Source: §1 · p. 1
• The law values context and weights by held-out code length saved and splits expected gain into stored information, prior repair, and decoding loss.The decomposition makes the two channels comparable term by term.Source: §1 · p. 1
• Capacity ceilings let one measured channel gain certify the choice, while an anytime-valid test compares channels on held-out queries with error controlled whenever it stops.The choice is against any model trained on the same data.Source: §1 · p. 2
• 25 of 64 certificates on a trained transformer and 36 of 96 on TabPFN, an occupancy network, and a stereo network agreed with the measured winner; the test selected TabPFN and a trained surrogate in every reported seed.On simulated streams with no true winner, it called one in at most 3.3% of streams, within its 5% limit.Source: §1 · p. 2
• Deleting context parts measures their value in any model, and adding them sequentially traces a learning curve; a feature-observing patch sharply lowers occupancy code length while patches missing the feature raise it.The paper also reports that evidence valuation recovers the Trace Budget adjudicator with AUROC 0.91.Source: §1 · p. 2

2. Learning from context

• In context, a frozen model builds a memory from the dataset and predicts from it; fine-tuning instead writes the dataset into the weights.A kriging learner without trained weights has only the context channel.Source: §2 · p. 3
• For a transformer, the context memory is its key–value cache, and teacher-forced passes measure its gain from the dataset.Deleting a part means rebuilding the memory from the rest and measuring the resulting ablation gain.Source: §2 · p. 3
• Under the clean-memory assumption, each memory is built only from its own data and exogenous randomness, and deletion rebuilds it from the remaining data without reusing state computed from the deleted part.The query is conditionally independent of the memory given the dataset.Source: §2 · p. 3

3. The law of in-context gain

• Expected gain conditions the resolution–reliability decomposition on the memory, separating stored information from what the decoder uses.When context fits the window, the fine-tune’s possible addition is capped by the context decoding loss.Source: §3 · p. 3
• The expected gain of either channel is bounded by the data’s total available value, G★, which is the query–data information plus prior repair.The storage loss is information about the query that the memory drops.Source: §3 · p. 3, Figure 1 · p. 3
• A matched deletion with the same form but no query-relevant content measures the form’s cost, so the difference from a relevant deletion isolates the contribution of content.The content difference can include changes to both information and decoding.Source: §3 · p. 4
• For a fixed context, the model’s own valuation is the code length it saves on its full-context continuation, and a deletion can be valuable when rebuilding from the rest changes a confident prediction.The valuation needs no labels and is defined separately for each context.Source: §3 · p. 4

4. Valuing context in any memory

• The evaluator needs a predictive distribution or claim probability, a deletion that permits memory rebuilding, and outputs scored with and without the deleted part.It was applied to occupancy, stereo, kriging, and language-model verifier memories.Source: §4 · p. 4
• Deleting 300 feature-observing points costs 17.66 bits per query, versus 0.001 bits for 300 matched points from unchanged geometry; an observing stereo view costs 1.62 bits per depth query, versus 0.55 bits for a non-observing view.The larger stereo loss occurs in all five scenes.Source: §4 · pp. 4–5
• Adding items one at a time traces the in-context learning curve, and a step that raises expected loss means storage and decoding losses together rose by more than the information the item adds.The valuation ranks support cars without labels; the top three code pressure better than four randomly drawn cars, −1.61 against −1.41 bits per value.Source: §4 · p. 5
• For a fixed binary claim, the evidence’s deletion gain supplies a bit budget that can be compared with the bits needed to raise the claim’s probability.The budget gap separates supporting from insufficient evidence with AUROC 0.91 for both Qwen2.5-7B and GPT-4.1-mini on VitaminC.Source: §4 · p. 5

5. Choosing between context and weights

• A context memory’s storage loss is at least the query information omitted from its state, while a fine-tune stores at most bP_train bits.For a context window, the omitted information is I(Q; D_ev | D_w); when the context fits the window and the memory keeps it, storage loss is zero.Source: §5 · p. 5

5.1. Ceilings and certificates

• Fine-tuning wins when its measured gain exceeds the context memory’s ceiling, and a capacity bound below the context gain plus prior repair certifies context at every N ≥ W.When the context fits and the memory keeps it, any fine-tune’s advantage is bounded by the context memory’s decoding loss.Source: §5.1 · p. 5
• A certificate needs one channel’s measured gain and the other’s ceiling, so it can settle the choice without running both channels.The fine-tune certificate requires a context that overflows the window; the context certificate requires its gain to exceed the fine-tune’s storage bound.Source: §5.1 · p. 6

5.2. Choosing without the true distribution

• For exchangeable units, prequential losses estimate learning curves by scoring each unit after building the memory from its predecessors.Their sums give prequential code lengths, but because learning curves can cross, the winner at a given size depends on the losses at that size.Source: §5.2 · p. 6
• An anytime-valid test compares held-out queries and controls its error whenever it stops, including when the trained model was trained from scratch.The stated guarantee follows from a nonnegative supermartingale whose threshold crossing rejects the no-worse hypothesis.Source: §5.2 · p. 6
• A state that reproduces in-context outputs within ε bits must store at least I(Ŷ_1:K; D) − ε bits about the context.The information requirement can be estimated from the model by resampling context content and scoring outputs under each variant.Source: §5.2 · p. 6

6.1. Certificates with exact information terms

• 25 of 64 cache–fine-tune comparisons were certified, and every certified call agreed with the measured winner.The fine-tune’s gain passes the cache ceiling from N = 128 with window 8 and N = 256 with window 16; when the context fits, the cache wins by 2.7 to 7.0 bits.Source: §6.1 · p. 7, Figure 3 · p. 7
• 36 of 96 certificates across TabPFN, occupancy, and stereo models were issued, and all 86 checks against measured winners agreed.TabPFN reading 1,024 rows was certified against every memory of at most 8 bits.Source: §6.1 · p. 7

6.2. Choosing without the true distribution

• The held-out-query test picks TabPFN on both tables and the trained surrogate on airfoil pressure in every seed, while GPT-2’s better cache or fine-tune is called in 61,948 of 64,000 streams.On airfoil pressure, the surrogate codes a value in 2.34 bits against the in-context learner’s 3.65.Source: §6.2 · p. 7, Figure 4 · p. 8
• Under three simulated nulls, the test calls a winner in 3.25%, 2.87%, and 3.31% of streams, within its 5% limit.For a 0.5-bit-per-query advantage, it calls correctly in every stream after a median of 127 queries.Source: §6.2 · p. 7

7. Related work

• Minimum description length treats learning as compression, while prequential codes score sequential forecasters.The paper extends this account from weights to context and separates bits stored from bits used.Source: §7 · p. 8
• Context attribution values prompt parts by ablation (Fernandes et al., 2021; Cohen-Wang et al., 2024), while deleting a part and rebuilding the memory gives its exact leave-one-out value in one rerun per part.A matched control separates the part’s content from its form.Source: §7 · p. 8
• Prior accounts interpret in-context learning as implicit Bayesian inference (Xie et al., 2022), a learned algorithm (Garg et al., 2022), or gradient descent and ridge regression run by attention (von Oswald et al., 2023; Akyürek et al., 2023; Mahankali et al., 2024).Information-theoretic analyses also express its error through information a context carries about a latent task (Jeon et al., 2024).Source: §7 · p. 8
• Proposition 13 writes softmax attention exactly as a weight write with query-dependent step sizes (He et al., 2022; Huang et al., 2026).The paper’s law prices that write and a fine-tune in the same bits.Source: §7 · p. 8
• Theorem 9 sets a lower bound on the divergence of a distilled state that stores a given number of bits about context (Askell et al., 2021; Snell et al., 2022).Proposition 6 and Theorem 8 address channel choice through a measured gain or held-out queries, respectively (Lampinen et al., 2025).Source: §7 · p. 8

8. Conclusion

• In expectation, code length saved by data in context or weights equals bits stored plus prior repair minus decoding loss.The law’s ceilings certified channel choices on a trained transformer, TabPFN, an occupancy network, and a stereo network, with every certificate correct.Source: §8 · p. 8
• The anytime-valid test chose between channels on tables, airfoil pressure, and GPT-2 within its error limit.The same evaluator valued context parts in occupancy, stereo, and kriging models and recovered a published evidence adjudicator for two language-model verifiers.Source: §8 · p. 9

A.1. Deleted-evidence exclusion

• Deleting evidence and rebuilding the memory values its contribution beyond the remaining context, while matched deletions distinguish relevant evidence from deletion cost alone.The deleted-evidence lemma states that the rebuilt memory carries no trace of the removed content, subject to its stated assumption.Source: §A.1 · p. 13

A.3. The fine-tune

• A fine-tune improves held-out-query compressibility when its training and query gradients align enough to exceed the curvature penalty, and cannot improve it below the corresponding negative threshold.The thresholds are stated for a bound on absolute curvature along the update step.Source: §A.3 · p. 14

A.4. The cache

• Per attention head, a cache is a query-dependent weight update, whereas a static weight edit uses frozen coefficients and leaves a transfer residual.The cache update has rank at most the smaller of the context size and head dimension; its coefficients are recomputed for each query.Source: §A.4 · p. 14

A.5. Attention without a cache

• A kriging learner’s support-point weights are attention-like, and deleting a support item and refitting performs the same intervention as rebuilding a cache.The described learner also selects settings by leave-one-out prediction within its forward pass.Source: §A.5 · p. 14

A.6. Capacity under noisy steps

• In-context learning wins when the data information exceeding fine-tuning capacity is greater than the context channel’s storage and decoding costs; a complementary condition identifies when fine-tuning wins.For a context memory that keeps its window, the fine-tuning condition compares evicted-data information with the storage and decoding terms.Source: §A.6 · p. 14

A.7. The lift budget; A.8. Estimating the storage budget; Impact Statement

• A channel’s probability lift is bounded by the bits it writes plus a data-independent drift term; a resampling estimator certifies storage budgets only below log2 M bits.The Impact Statement says these methods measure what context supplies and add no new capability to evaluated models.Source: Impact Statement · p. 9, §A.7 · p. 15, §A.8 · p. 15

A.9. The transfer integral

• A fine-tune transfers query bits only as it compresses its training data; a bounded transfer ratio can rule out matching the context memory, and a negative ratio indicates interference.The initial ratio costs one backward pass per query and one for the training data; orthogonal gradients give an initial ratio of zero.Source: §A.9 · p. 15

A.10. A model where every term is exact

• One optimally sized gradient step has exactly the same excess risk as one linear-attention layer at every n and d.In the proportional limit, the best step size is η*=τ̄^2/(τ̄^2(1+γ)+σ^2γ).Source: §A.10 · p. 16
• At ρ=4 and adapter dimension k=d/4, the adapter ceiling is 14% of the cache’s ceiling.The comparison is for a frozen random down-projection.Source: §A.10 · p. 17
• At ρ=4, matching an ideal cache holding N examples requires n*/N=1.01–1.06 for five or more gradient steps, versus 1.2–3.8 for one step.These crossover ratios span N/d from 1/4 to 4.Source: §A.10 · p. 17
• At d=200 with 300 repeats per point, simulations reproduce the closed forms with a largest gap of 0.005 bits across Figure 5’s points.Figure 5 compares closed-form lines with simulation dots at ρ=4.Source: §A.10 · p. 17

B.1. Deleted-evidence exclusion

• The deleted-evidence lemma follows because rebuilding from the remaining inputs uses transition kernels whose conditional laws do not depend on deleted content.The admissibility discussion distinguishes recomputed transformer states from cached states whose provenance depends on removed inputs.Source: §B.1 · p. 17

B.2. The law and its corollaries

• The expected gain identity follows by applying the chain rule to query information in the data and memory, separating stored information from decoding loss.The proof appendix also shows that prior repair cancels when comparing matched deletions.Source: §B.2 · p. 17

B.4. The fine-tune; B.5. The cache

• The fine-tune proof integrates the change in query log probability along gradient flow, while the cache proof decomposes attention into context and prefix contributions.For a discrete update, the appendix uses Taylor’s theorem with an integral remainder.Source: §B.4 · p. 18, §B.5 · p. 18

B.6. Capacity and crossover; B.7. Ceilings and certificates

• The capacity proof bounds context storage by information in the window and fine-tune storage by its capacity; the resulting inequalities support channel certificates.When the context fits and preserves its window, its storage term is zero.Source: §B.6 · p. 18, §B.7 · p. 18

B.8. Learning curves, the anytime-valid test and the budgets

• The anytime-valid test proof constructs a nonnegative supermartingale under the null and applies Ville’s inequality to control the probability of crossing its threshold.The argument conditions on held-out queries and allows stopping at any time.Source: §B.8 · p. 19

C.1. One evaluator for any memory

• The evaluator fixes claims before deletion, rebuilds memory for each deletion, and compares relevant evidence with matched controls; spatial averages use queries or scenes as statistical units.Controls include equal-size deletions from unchanged geometry and source views that do not observe the queried point.Source: §C.1 · p. 21

C.2.1. A FEATURE GRID: 17.7 BITS PER QUERY FROM A LOCAL SHAPE FEATURE

• 17.66 bits per query come from deleting the 300 points that distinguish a local shape feature; an equal-size unchanged-geometry deletion costs 0.00111 bits.The relevant deletion drops mean occupancy probability from 0.9827 to 0.0014, and all 7,473 confident feature claims fall below 0.95.Source: §C.2.1 · pp. 21–22

C.2.2. DONOR EVIDENCE COUNTS WHERE ITS GEOMETRY AGREES

• Donor evidence preserves confidence when its geometry agrees with the claim: 89.88% of agreeing-donor claims remain confident, versus 2.47% for disagreeing donors.In the coherent-object experiment, the target’s observations yield 0.9587 mean occupancy probability at target-only interior coordinates, versus 0.0351 with donor observations.Source: §C.2.2 · p. 22

C.2.3. A COST VOLUME: CAMERAS THAT SEE THE SURFACE CARRY THREE TIMES THE BITS

• An observing source view costs 1.6204 bits per query to delete, versus 0.5452 bits for a non-observing view; its claim-probability loss is 9.52 percentage points.The larger loss occurs in all five scenes, and is 4.5 times larger for correct than incorrect depth intervals.Source: §C.2.3 · p. 23

C.2.4. REQUIRED AND SUPPLIED BITS

• Relevant deletions supply more bits and move more claims across the target than matched controls; the feature item supplies 17.66 bits against a 16.93-bit requirement.An observing view supplies 1.62 bits against 1.27 required, while a support car supplies 0.23 against 0.74 required.Source: §C.2.4 · pp. 23–24

C.2.5. ATTENTION WITHOUT A CACHE: NATIVE INFERENCE IS PART OF THE INTERVENTION

• Native inference cuts mean relative L2 pressure error from 0.5276 to 0.2992, a 43.3% reduction, with all eight cars improving.The mean single-car value falls from 0.79 to 0.23 bits under native inference, which passes the full query geometry rather than only 512 evaluation sites.Source: §C.2.5 · p. 24

C.2.6. A KEY–VALUE CACHE: THE BINARY CASE IS EVIDENCE ADJUDICATION

• The budget gap separates supporting from insufficient evidence with AUROC 0.9126 for Qwen and 0.9143 for GPT-4.1-mini.For supporting versus all unsupported evidence, AUROC is 0.8990 and 0.9297, respectively.Source: §C.2.6 · pp. 24–25

C.2.7. LEARNING CURVES IN CONTEXT

• One observing source view codes depth better than all four views together in scenes 4, 29, and 48; valuation-ranked support-car orders outperform random orders in six, five, and six of eight cars at sizes three, four, and five.Source: §C.2.7 · p. 25

C.4. The choice across domains in full

• On both OpenML tables, TabPFN encodes held-out labels in fewer bits than XGBoost in every seed; on airfoil pressure, the trained surrogate wins in every seed.With 32 designs, the in-context learner is selected in two of three seeds.Source: §C.4 · p. 27

Learning curves from the context

• For Shakespeare, prequential increments agree with held-out learning curves within at most 1.97 standard errors across all eight size bins and both channels.Their summed code lengths are within 0.1% for the cache and 1.5% for the fine-tune of held-out-curve sums.Source: §C.5 · p. 28, Figure 11 · p. 29

An anytime-valid choice

• Across Shakespeare and document-recall streams, the anytime-valid test calls the better memory in 61,948 of 64,000 query streams; it calls the losing memory in four.On null streams, it reaches its threshold in at most 0.66% of 10,000 streams for any of 32 comparisons.Source: §C.5 · p. 28, Table 12 · p. 29, Table 13 · p. 30

A lower bound on GPT-2’s decoding loss

• With four context units, GPT-2’s online fine-tune beats its cache by 0.35 to 0.55 bits per token on held-out passages, bounding the cache’s decoding loss.The held-out passages are independent of those in context, so the passage attributes the shortfall to unrealized prior repair.Source: §C.5 · p. 29

The storage budget

• GPT-2’s outputs carry 5.14 bits about the 12-bit context table; replacements storing less than 5.14 bits miss the cache by at least the difference in stored bits.GPT-2 reading compiled prefixes adds 3.6 to 25.6 bits of decoding loss to every replacement short of the full table.Source: §C.5 · pp. 29–30, Figure 11 · p. 29

C.6. Certificates on pretrained models outside language

• 36 of 96 certificates were issued, and all 86 checks agreed with the measured winners across three pretrained models.TabPFN's certificate for learning in context fires from 64 rows; the occupancy and stereo networks also receive certificates under stated memory and context-window bounds.Source: §C.6 · pp. 30–31

D.1. Geometry: local features and natural objects

• Geometry pairs differ by a raised feature versus a matched recess, and the relevant deletion removes the distinguishing observations while leaving identical paired inputs.The pretrained model is evaluated without fine-tuning.Source: §D.1 · p. 31

D.2. Multi-view preprocessing, claims, and visibility

• Stereo visibility is determined by projecting reference surface points into source views and comparing them with independently supplied source depths.Unknown views are not treated as non-observing, and the paired oracle-inside cohort contains 11,808 query locations across 25 references.Source: §D.2 · p. 32

D.3. Kriging learner and native inference

• The kriging learner selects its inference settings by leave-one-car-out prediction, and native inference improves all eight query cars.It needs at least two support cars, so every deletion leaves at least two.Source: §D.3 · pp. 32–33

D.4. Language adjudication

• On 797 successful GPT-4.1-mini examples, sufficiency AUROC is 0.9152 and full-grounding AUROC is 0.9318.Three requests returned content-filter errors and counted as flagged.Source: §D.4 · p. 33

D.5. Decision test

• The decision test compares cache and fine-tune certificates using paired held-out-task margins, issuing a certificate when its 95% interval lies above zero.The 64 evaluation tasks are shared across channels and data sizes.Source: §D.5 · pp. 33–34

D.6. GPT-2 experiments

• GPT-2 experiments compare cache and fine-tune predictions on Shakespeare streams, document-recall spans, and a storage-budget task.For the Shakespeare test, both directions are run on each stream, and the first to reach the stopping threshold makes the call.Source: §D.6 · pp. 34–35

D.7. Tables and airfoil pressure

• The table and airfoil comparisons fix settings before scoring queries and use held-out data to compare TabPFN with XGBoost and an in-context learner with a trained surrogate.The airfoil evaluation uses 254 query designs from groups absent from earlier training, calibration, and test sets.Source: §D.7 · p. 35

D.8. Learning curves in context

• Context learning curves score each item after rebuilding memory from the preceding subset; the sweeps use varied orders for occupancy, stereo, and kriging inputs.Kriging orders include one ranked by the learner's own full-context gains.Source: §D.8 · pp. 35–36

D.9. Numerical conventions

• The scorer computes log probabilities in a stable log domain, retains infinite gains, and checks numerically that claim-level bits do not exceed gain.Source: §D.9 · p. 36

E. Extended related work

• The paper treats context value as saved code length and identifies expected gain with stored information plus prior repair minus decoding loss.It relates this account to minimum-description-length and prequential coding.Source: E. Extended related work · p. 36
• The related-work discussion places the cache as a query-dependent weight write and presents anytime-valid code-length comparison as a choice between context memory and fine-tuning.It connects these results to softmax attention (He et al., 2022; Huang et al., 2026) and forecast testing (Ramdas et al., 2023).Source: E. Extended related work · pp. 36–37

• The supplementary README maps each script to the paper section, figure, and table it reproduces.Source: F. Reproducibility · p. 37
• Saved laws and stopping traces replay the Section 6.2 comparisons without model weights, while scripts recompute the airfoil and context learning curves.Repeating model executions requires the original datasets and pretrained weights.Source: F. Reproducibility · p. 37