PaperlayerView exact revision on arXiv

Source-linked AI summary

Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel

Sietse Schelpe

arXiv:2607.14431v1cs.CLcs.AIcs.LGcs.PF

TL;DR

Repeated context recomputation and retraining make capability and inference costly. The paper uses verified byte-exact KV-state grafts to reuse knowledge in a frozen model, reporting higher benchmark performance, exact restoration, and substantially lower decoding cost.

  • Problem

    Capability gains require retraining or scaling, while inference repeatedly recomputes and discards the same context, compounding costs.

  • Method

    The system verifies solutions, stores them as byte-exact KV states, and grafts selected states into a frozen model for recurrence and held-out transfer.

  • Results

    93.3% on post-cutoff AIME was achieved from 80.0% by a frozen 12B, while graft restoration produced zero argmax disagreements over 50 samples.

  • Takeaways & Limitations

    Verified KV-state reuse can make a frozen small model more capable while reducing repeated decoding work and avoiding additional accelerator memory for expanded usable context.

  • Takeaways & Limitations

    Transfer is reliable only when the cached program is parametric, and byte-exactness is restricted to the same architecture and configuration.

Abstract

from arXiv · show

We report a way to make a frozen small language model both more capable and dramatically cheaper at once, without changing any weights. Verified knowledge is deposited once as a byte-exact key-value (KV) state artifact and later restored, by graft, into a fresh inference context. The restore is bit-exact: under a pinned deterministic configuration, the grafted logits are byte-for-byte identical to a fresh computation (SHA-256 equality), with zero KL divergence and 100% argmax agreement over fifty samples. We show that own-position graft is the unique numerically exact operating point on a model with floating-point rotary encoding, and we verify byte-exactness on two model scales (12B, 31B) and two GPU targets, one through a pre-registered replay. On AIME 2025, a frozen Gemma-4-12B moves from 80.0% to 93.3% once a verified solution library is grafted, above its own 77.5% and its 31B sibling's 89.2% published anchors. On the recurring case, eight problems the base model never solves within a 401,026-token budget are answered from cached verified solutions in 61 total decode tokens, a factor of 6,574 fewer tokens and about 8,700x less energy; the capability claim proper rests on held-out transfer (7 of 7 at 31B). The same byte-exact store widens usable context from 32,768 to 2,854,766 tokens at zero extra accelerator memory, and moves byte-identical between machines of the same architecture. We describe the system at the behavior level; the engine is proprietary, and every reported number is backed by committed input and output hashes so the scoring can be re-checked without it.

1 Introduction

The paper argues that a frozen small model can become both smarter and cheaper by storing verified knowledge as byte-exact KV state and restoring it by graft, avoiding retraining and repeated context recomputation. It presents Taliesin and Galahad as a measured, portable, persistent KV-state system whose exactness is verified at the logits and distribution levels.

  • Two costs compound: improving capability requires retraining or fine-tuning, while using capability requires recomputing context on every call.
  • Verified KV state can be captured once, written to disk, and grafted into a different context without recomputation or loss.This would let deposited verified state add knowledge without a gradient step.
  • Bit-exact restoration is verified by byte equality of resulting logits and zero divergence of the resulting distribution.The paper also identifies the precise operating conditions required for exactness.
  • Taliesin is the byte-exact graft mechanism, while Galahad is the verify-then-cache learning loop built on top of it.
  • The central thesis is that a frozen small model becomes smarter and cheaper simultaneously by adding verified knowledge as byte-exact KV state and retrieving it by graft.The study measures this claim on real Gemma-4 models and benchmarks, against published anchors, while reporting honest negatives.

2 Background and Positioning

This section positions KV-cache persistence and sharing as established infrastructure while distinguishing approximate, ephemeral reuse from the paper’s exact, durable verified-knowledge flywheel. It also frames comparisons against published benchmark anchors rather than unaffordable local reruns.

  • Related work: Persisting, offloading, encoding, and sharing KV caches across engine instances are established capabilities in systems including LMCache, llm-d, KServe, and CacheGen.The paper identifies these surrounding capabilities as commodity infrastructure.
  • Related work: Approximate, ephemeral reuse differs from exact, durable reuse: native prompt caches remain process-tied, while token retrieval incurs injection costs.The section presents this distinction as central to the paper’s positioning.
  • Contribution: The verified-knowledge flywheel solves and verifies a problem once, freezes its solution as a byte-exact KV block, then grafts it instead of re-deriving.The block remains on disk at zero accelerator memory and can be copied to a fresh server for byte-identical grafting without re-solving.
  • Evaluation framing: Comparisons use published benchmark anchors rather than local base-model reruns when fair reruns are unaffordable, comparing a frozen small model plus Taliesin with larger-model strength.The authors explicitly characterize this as the comparison framing rather than a head-to-head local re-run.

3 Methodology

The methodology defines byte-exact KV grafting as raw-byte equality under pinned deterministic numerics, then evaluates a verification-gated flywheel that stores and routes persistent solution blocks. The system is described behaviorally because its internal engine and benchmark suite are proprietary.

  • System and measurement scope: Internal state capture, restoration, storage, deduplication, and hashing mechanisms remain proprietary, and reproducing generation requires the separately licensed Merlin/Taliesin benchmark suite.The paper states only the behavioral contract, preserved invariants, and numerical facts needed to interpret measurements.
  • Exactness guarantee: Byte-exactness means fresh and grafted logit vectors have identical raw bytes, verified directly with SHA-256 rather than merely matching norms.The guarantee also requires KL divergence of zero and identical argmax outputs on every trial.
  • Exactness guarantee: Pinned deterministic settings establish a zero self-comparison floor, making grafted outputs indistinguishable from repeated fresh computations under matched conditions.Measurements use GGML_DETERMINISTIC=1 and CUBLAS_WORKSPACE_CONFIG=:4096:8, with build and runtime settings recorded for audit.
  • Flywheel protocol: Each difficult problem is solved with extra inference, externally verified, and deposited as a persistent KV block; only verified solutions enter the store.For AIME, verification executes the generated program and checks its known answer; for LiveBench, it matches ground truth.
  • Retrieval regimes: Recurrence performs exact lookup of the relevant grafted answer, while transfer routes a single block and adapts its verified method to a structurally new problem.Transfer uses independently re-verified AIME variants or a temporally held-out LiveBench split with zero question-id overlap.
  • System and measurement scope: The reported system deposits each verified solution separately and routes only the relevant block, replacing an earlier flat prefix that confused the model.The study reports local measurements across CPU, CUDA, Gemma-4-12B GPU, and H100 SXM environments, with idle draw and token counts recorded where applicable.

4 Empirical Results … 4.3 The cost engine: an 85.6x prefill subsidy

The experiments establish that own-position KV grafting reproduces fresh computation exactly under deterministic conditions, while repositioning is numerically inexact because floating-point rotary encoding is position-sensitive. This lossless reuse yields an 85.6x prefill speedup by evaluating one token instead of 11,994.

  • 4.1 The graft is bit-exact at its own position (KL = 0, SHA-equal): 50 samples show zero median and p99 KL divergence, zero graft failures, and zero argmax disagreements between grafted and fresh logits.The Mann-Whitney test gives p = 1.0000 with Cliff’s delta 0.0000.
  • 4.1 The graft is bit-exact at its own position (KL = 0, SHA-equal): 5 of 5 independent trials produce identical SHA-256 hashes for grafted and fresh logit bytes.On Gemma-4-12B, 50 samples likewise show zero graft failures, zero argmax disagreements, and KL at the floating-point floor near 10^-27.
  • 4.1 The graft is bit-exact at its own position (KL = 0, SHA-equal): Runtime migration preserves the guarantee: KL remains 0 and SHA remains equal on a production-scale Gemma-4-12B under the modern runtime.The result was re-measured after a major inference-runtime refactor.
  • 4.2 The numerical regime of positional graft: own-position is the unique exact point: Re-basing a captured block to offsets M ∈ {8, 128, 1024, 4096} yields roughly 0.015 KL and a handful of argmax flips versus fresh prefills.Own-position grafting remains byte-exact; fresh-prefill controls show the residual reflects the model’s own position sensitivity.
  • 4.2 The numerical regime of positional graft: own-position is the unique exact point: 32-bit floating-point rotary encoding makes own-position graft the unique numerically exact regime; other absolute offsets cannot attain byte-exactness against their fresh references.Different absolute angles round differently, producing an irreducible residual near 0.01 after network amplification.
  • 4.3 The cost engine: an 85.6x prefill subsidy: 85.6x speedup reduces an 11,994-token cold prefill from 1,547.3 ms to 18.1 ms by reusing the cached state and advancing one token.The reuse is lossless under the exactness result above.
  • 4.3 The cost engine: an 85.6x prefill subsidy: 83.7x speedup reduces a fresh 5,569-token prefix from 1,087 ms cold to 13 ms warm after full-size KV resumption and slot pinning.Full-size KV for windowed layers raises memory use from 19.7 to 25.5 GB at 64k context.

4.4 Widening the usable context 87x at zero extra memory · 4.5 The flywheel makes a 12B smarter: 80% to 93.3% on post-cutoff AIME

Byte-exact KV blocks widen usable context from 32,768 to 2,854,766 tokens without extra accelerator memory, while a grafted verified library raises frozen Gemma-4-12B’s AIME 2025 score from 80.0% to 93.3%. The same approach also supports cheaper inference, with cached knowledge lifting the lean run from 76.7% to 90.0% for roughly 5.6% more tokens.

  • 4.4 Widening the usable context 87x at zero extra memory: 2,854,766 stored tokens—87.1x the 32,768-token serving window—fit with 0 extra accelerator memory.The store used 88 persistent blocks and occupied 40.6 GB on disk; peak accelerator memory remained 25,595 MB.
  • 4.4 Widening the usable context 87x at zero extra memory: 7 of 7 needles were retrieved correctly across depths from 0 to 2.82M tokens.Restoration cost was 0.29 s plus approximately 35 newly prefilled tokens, constant with depth.
  • 4.4 Widening the usable context 87x at zero extra memory: A negative control confirmed that querying a distant block moves the live window rather than retaining 2.8M tokens in attention.A native 2.85M-token context would require KV memory beyond the 32 GB card, whereas the disk store reaches any point by restoration.
  • 4.4 Widening the usable context 87x at zero extra memory: 6 of 6 agent tasks routed correctly after restoring a 160 MB byte-identical KV state captured from a 464-token skill manual.The demonstration shows persistence to disk and restoration into a fresh process, unlike a native in-memory cache.
  • 4.5 The flywheel makes a 12B smarter: 80% to 93.3% on post-cutoff AIME: 80.0% (24 of 30) was the frozen Gemma-4-12B score on AIME 2025 before grafting the verified library.This exceeded the model-card anchor of 77.5%; the confidence gate marked 22 problems confident and eight unsure.
  • 4.5 The flywheel makes a 12B smarter: 80% to 93.3% on post-cutoff AIME: 93.3% (28 of 30) followed grafting, with six of the eight unsure targets recovered from an eight-entry verified library.The library was approximately 4,571 tokens and captured as a 441,583,860-byte KV state; the two misses were #11 and #14.
  • 4.5 The flywheel makes a 12B smarter: 80% to 93.3% on post-cutoff AIME: 90.0% (27 of 30) was reached at 4,360 decode tokens per problem and 7.0 Wh per problem in the lean code-routing run.This configuration differs from the confidence-gated 93.3% system; cached knowledge raised the lean score from 76.7% for roughly 5.6% more tokens.

4.6 The flywheel makes a 12B cheaper: 6,574x fewer tokens on recurrence · 4.7 Transfer: the same store generalizes to new data, within a stated boundary

The flywheel solves eight recurring problems with verified cached states using 61 decode tokens instead of 401,026, while transfer to seven independently verified variants succeeds only when cached programs remain parametric in swapped numbers. The results separate cheap recurrence from bounded autonomous adaptation, with routing and energy-measurement caveats explicitly reported.

  • 4.6 The flywheel makes a 12B cheaper: 6,574x fewer tokens on recurrence: 61 decode tokens solve all 8 recurrence problems, versus 401,026 tokens for base best-of-5, which solves 0 of 8.The result is reproduced across four independent runs using the frozen model plus persistent solution-state.
  • 4.6 The flywheel makes a 12B cheaper: 6,574x fewer tokens on recurrence: 6,574x fewer decode tokens, roughly 3,000x to 8,700x less energy, and ~1,700x faster yield 8/8 versus 0/8.The integrated measurement reports 0.053 Wh over 2.8 seconds at mean 186 W.
  • 4.6 The flywheel makes a 12B cheaper: 6,574x fewer tokens on recurrence: 0.145 Wh is the conservative energy bound at the sampler’s resolution limit, reducing the energy advantage to roughly 3,000-fold.The bound uses mean sampled power multiplied by wall time for the 2.8-second run.
  • 4.6 The flywheel makes a 12B cheaper: 6,574x fewer tokens on recurrence: 5 of 8 queries are recalled correctly with all blocks present, whereas routing to one correct per-block block removes cross-entry interference.The flat-prefix setup returned another problem’s answer; all eight recurrence queries retrieved their correct blocks under routing.
  • 4.7 Transfer: the same store generalizes to new data, within a stated boundary: 5/7 variants transfer successfully, with routing 7/7, when the frozen 12B adapts cached programs to independently verified swapped-number problems.The seven variants were cross-checked three ways and rejected if their answers equaled the originals.
  • 4.7 Transfer: the same store generalizes to new data, within a stated boundary: #28 over-adapts range(4) to range(2024), causing exact-fraction blowup and timeout; #11 hard-codes sqrt(185), while the variant needs sqrt(697).The latter cannot transfer by number substitution, so the model rewrites it and times out.
  • 4.7 Transfer: the same store generalizes to new data, within a stated boundary: 5 of 7 transfer succeeds for parametric cached programs, while baked-in problem-specific constants can force autonomous re-derivation and failure.Mechanical literal substitution makes the cached program solve 7 of 7; no clean same-structure variant existed for #10.

4.8 Composition: sequential prefill composes, isolated stitching does not · 4.9 Cross-architecture scale-up: a frozen 31B on an H100 · 4.10 Cross-architecture byte-exactness: a pre-registered B200 replay

Sequential prefill composes cached blocks, whereas isolated stitching fails because the later block never attended to the earlier one. The method also transfers to a frozen 31B on Hopper and passes a pre-registered B200 byte-exactness replay across architectures.

  • 4.8 Composition: sequential prefill composes, isolated stitching does not: KL 13.85 and 50 of 50 argmax flips show that stitching independently captured final states fails against a fresh concatenated prefill.The failure is attributed to causal starvation: block B was prefilled in isolation and never attended to block A.
  • 4.9 Cross-architecture scale-up: a frozen 31B on an H100: 100% held-out transfer (7 of 7) shows the frozen Gemma-4-31B correctly adapts every cached method to new numbers.The transferred methods include cases that the local 12B got wrong.
  • 4.9 Cross-architecture scale-up: a frozen 31B on an H100: 100% on the full-30 system is a flywheel-system number: eight hardest problems use verified cached solutions, while twenty-two are solved live.The clean unseen-generalization result is the 7 of 7 transfer score.
  • 4.9 Cross-architecture scale-up: a frozen 31B on an H100: 89.2% is the 31B model-card anchor that both the full-30 system and the clean transfer result beat.The run’s total cloud spend was roughly 8 to 12 euros.
  • 4.9 Cross-architecture scale-up: a frozen 31B on an H100: 71.7% held-out transfer (43 of 60) on LiveBench fell to 56.7% when all blocks were merged, confirming single-block routing.The held-out split used older releases as the library and the newest release without question-id overlap.
  • 4.10 Cross-architecture byte-exactness: a pre-registered B200 replay: Zero graft failures, zero of fifty argmax disagreements, and SHA byte-equality on ten of ten trials passed the pre-registered B200 replay criteria.Exactness was reported for Gemma-4-12B and 31B on consumer Blackwell sm_120 and datacenter Blackwell B200 sm_100.
  • 4.10 Cross-architecture byte-exactness: a pre-registered B200 replay: 19 times smaller position sensitivity on the 31B B200 replay still produced divergence, with a median near 7 × 10^-4 versus 1.4 × 10^-2 on the 12B.The result confirmed that position sensitivity reappears across GPU targets rather than being specific to the original silicon.

4.11 Portability: the learned state is a carriable, byte-exact asset (within an architecture) · 4.12 Systems behavior: router misrouting and disk paging, measured · 4.13 Honest negatives (recorded, not hidden)

The learned state transfers byte-exactly between same-architecture machines and preserves savings under measured routing and disk paging. The paper also records explicit limits: caching is useful only for missing knowledge, can harm when it overrides a better method, and is not universally byte-exact or effective.

  • 4.11 Portability: the learned state is a carriable, byte-exact asset (within an architecture): 8 of 8 moved AIME blocks functioned on a freshly built same-architecture server, with destination SHA-256 digests equal to the source digests.The blocks were copied without re-depositing, so the first machine’s one-time add-knowledge cost was inherited on the second.
  • 4.12 Systems behavior: router misrouting and disk paging, measured: 15 of 15 in-library routing cases were correct, with a Wilson 95% confidence interval of about 79.6% to 100%.Recurrence uses exact lookup; only transfer uses one-shot classification, and the raw pick was recorded before fallback.
  • 4.12 Systems behavior: router misrouting and disk paging, measured: 19 ms cache-hit prefill remained near constant across prompt lengths after disk-to-accelerator restore, indicating state reuse rather than recomputation.The comparison was against fresh re-prefill across prompt lengths.
  • 4.12 Systems behavior: router misrouting and disk paging, measured: 96 to 748 ms re-prefill time grew across an eightfold token increase, while disk-to-accelerator load rose only from 58 to 86 ms.Paging overhead therefore did not consume the prefill subsidy; the subsidy widened with context length.
  • 4.13 Honest negatives (recorded, not hidden): 216 of 216 examples were solved by both the learning arm and frozen control with identical time, leaving no failure-room for caching to save.This null result occurred on an auto-generated task family that the 2026 12B solved first-try.
  • 4.13 Honest negatives (recorded, not hidden): 55 points of accuracy were lost and tokens multiplied tenfold when a cached Chinese-remainder procedure forced a worse method than the model’s default.The resulting rule is to cache verified knowledge the model lacks, not re-prescribe methods it already has.
  • 4.13 Honest negatives (recorded, not hidden): Byte-exact portability is limited to machines within one architecture: H100-to-H100 transfer was functionally validated, while cross-architecture raw-byte equality is not guaranteed.Byte-level grafting was measured on two Blackwell targets; floating-point accumulation differences can change bytes across architectures.
  • 4.13 Honest negatives (recorded, not hidden): 5 of 20 LiveBench multi-exemplar blocks achieved recurrence on the 31B, versus 8 of 8 for AIME one-solution blocks.The limitation reflects weak recurrence on complex multi-exemplar blocks, not failure of the caching mechanism itself.

5 Discussion

The discussion argues that verified KV-cache grafting can reduce reasoning cost without relying on larger models, because byte equality makes cached knowledge trustworthy. It also defines transfer, recurrence, portability, and evidence boundaries, distinguishing system scores from held-out generalization.

  • Cost: 90.0% at 4.4k tokens per problem versus 76.7% at 25k tokens per problem makes cached verified knowledge roughly fifty times cheaper per point.Sampling-and-voting pays its full cost on every query, whereas a verified deposit pays once and can be reused.
  • Model size and cost: 2.44M tokens versus 768k tokens showed the large mixture-of-experts model was more verbose and failed to finish half its attempts while the 12B model had zero truncation.The discussion concludes that token cheapness does not guarantee lower compute per solved problem for hard reasoning under a fixed memory budget.
  • Exactness: Byte equality removes doubts that capability gains or recurrence results arise from approximation drift, treating the grafted state as the model having read the knowledge.This exactness is the basis for using a knowledge deposit as a unit of learning.
  • Limitations: Transfer requires cached programs to be parametric in changing quantities, while recurrence is weak for complex multi-exemplar blocks and cached procedures can harm already-solved tasks.Byte-exactness is limited to the same architecture and configuration, and positional relocation is unattainable on this model.

6 Conclusion

The conclusion presents byte-exact KV-state grafting as a way to make a frozen small model smarter and cheaper without training or additional accelerators. It reports exactness under pinned deterministic conditions and frames verified-state retrieval as an economic alternative to recomputation and larger scale.

  • Byte-exact KV-state grafting targets making a frozen small model smarter and cheaper without training it.
  • SHA-256 equality, KL = 0, and 100% argmax verify the mechanism under a pinned deterministic configuration.The passage also identifies the operating point as the only one attaining exactness with floating-point rotary encoding.
  • Verified knowledge added as exact state is cheap to store, while retrieval is far cheaper than large-model recomputation without changing weights.The conclusion characterizes this as an economic alternative to answering every gap with more scale.

Reproducibility statement

The reported results are reproducible through committed hashes, raw generations, and solver code that enable scoring checks without the proprietary engine. A pre-registered cross-architecture replay further specified its source-tree hash, pass criteria, and falsifiable prediction, while the engine and state-management machinery remain closed.

  • Reproducibility statement: SHA-256 digests cover exact input datasets, runner scripts, output result files, and raw generations.Inputs and scripts were hashed before each run; outputs and generations were hashed on exit.
  • Reproducibility statement: The cross-architecture replay pre-registered its source-tree hash, pass criteria, and falsifiable prediction before execution.Result files include per-problem raw generations and solver code sufficient to re-check scoring without the proprietary engine.
  • Reproducibility statement: The engine, state capture and restore, and deposit-path deduplication and hashing machinery are closed and available only under separate terms.These components are not required to audit the honesty of the reported results.

Author contributions and inventorship

Sietse Schelpe is the paper’s sole author and inventor of its Merlin, Taliesin, and Galahad systems, and he led the entire research process. The dated, hashed record underlying the results is retained and available under the reproducibility statement’s terms.

  • Author contributions and inventorship: Sietse Schelpe is the sole author and inventor of the Merlin engine, Taliesin graft mechanism, and Galahad flywheel.He conceived, designed, built, ran, analyzed, and wrote the work.
  • Author contributions and inventorship: The dated, hashed experimental record underlying every result is retained and available under the reproducibility statement’s terms.
Loading 2607.14431v1…