PaperlayerView exact revision on arXiv

Source-linked AI summary

KV-Lingo: Learning KV-Cache Translators with Distillation

Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin

arXiv:2609.32610v1cs.CLcs.LG

TL;DR

Model-specific KV caches make switching over shared context costly because the target must reprocess the prefix. KV-Lingo translates caches between frozen models using linear maps trained by distillation. The approach preserves strong downstream performance and reduces time to first token after switching.

  • Problem

    Because KV caches are model-specific, switching models over shared context requires the target to re-encode the prefix or maintain a cache for every possible model.

    Source: §1 · p. 1

  • Method

    KV-Lingo trains token-wise linear cache translators by distilling the target model’s predictions from its native cache onto predictions from the translated cache.

    Source: §1 · pp. 1–2

  • Results

    4.8× to 29× faster time to first token is reported for model switching, while translated caches perform as well as or better than the smaller model on most tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Cache translation provides a primitive for dynamic routing and hand-off between language models without re-prefilling the shared prefix.

    Source: §5 · p. 10

    The setting keeps source and target models frozen and trains only the translator; jointly training models for compatible caches is outside the paper’s scope.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens' key and value representations. We train these maps using distillation, minimising the divergence between the target model's predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6x already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29x at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.

1 Introduction

Model-specific KV caches make switching models over shared context costly because the target must re-encode the prefix. KV-Lingo translates caches between frozen models, preserving strong downstream performance while reducing switching latency.

1 Introduction

  • Re-encoding shared context at each model switch duplicates computation, with the cost growing with prefix length.Caches are model-specific, so the target otherwise builds its own cache before decoding.Source: §1 · p. 1
  • KV-Lingo uses token-wise linear maps to translate a source cache into a target-readable cache, trained by distillation against the target’s predictions.The maps are the only trained component; the models remain frozen.Source: §1 · pp. 1–2
  • Across model pairs spanning 0.6B to a 30B mixture of experts, translated caches perform as well as or better than the smaller model on most tasks.The evaluation includes language modelling, multi-turn chat, and long-context understanding.Source: §1 · p. 2
  • 4.8× to 29× faster time to first token is reported for model switching with KV-Lingo, depending on model and prefill length.Translation replaces a full target prefill with a linear translation and one decoding step.Source: §1 · p. 2

2 Problem setting

The paper frames cache translation as a way for one frozen model to hand a prefix to another without rebuilding the target cache. It describes both transfer directions and deployment settings where their trade-offs matter.

2 Problem setting

  • The KV cache stores past keys and values, reducing each decode step to O(n) in context length, but its model-specific shape and coefficients make direct transfer degrade generation.At long context, the cache dominates serving memory cost.Source: §2 · p. 3

2.1 Cache translation

  • A translator maps a source cache into a target-readable cache so the target can continue generating from the same prefix.The source and target models are frozen; only the translator is trained, while joint training for compatible caches is outside scope.Source: §2.1 · p. 3
  • Translation need only match the target’s continuation distributions, not reconstruct its cache entrywise, and must cost less than re-prefilling to pay off.Cache reconstruction error and continuation quality can disagree sharply.Source: §2.1 · p. 3

2.2 Deployment scenarios

  • Small-to-large transfer lets a faster model prefill before handing off to a stronger model; large-to-small transfer lets a cheaper model decode from the larger model’s processed context.The latter is described for agent pipelines and privacy settings; an on-device implementation would also need to weigh cache-transfer communication costs.Source: §2.2 · p. 3
  • A dense model can prefill for a mixture-of-experts target, which then decodes from the translated cache and may provide better quality through its greater capacity.The paper also describes directly using the smaller model’s cache by merging translators with queries, but tests this only as a proof of concept requiring post-RoPE and sparse translators.Source: §2.2 · p. 4

3 KV-Lingo: linear translators trained with self-distillation

KV-Lingo uses structured linear maps to translate source-layer keys and values into each target layer, with source-layer selection for differing depths. Its two-stage training first fits cache mappings, then distills the target’s continuation predictions.

3.1 Linear translator family

  • Each target layer maps keys and values from selected source layers, using the counterpart layer when source and target depths match.For cross-layer pairs, source-layer selection can use relative-depth matching, greedy matching, or the best single-layer fits; larger source-layer bands trade speed for quality.Source: §3.1 · pp. 4–5
  • Linear maps are chosen for speed, support closed-form initialization, and nearly match an MLP’s performance; mapping values alone halves parameters with negligible performance cost.Head-wise maps are faster but less expressive than head-mixing maps, which use more parameters.Source: §3.1 · pp. 4–5
  • Pre-norm and post-RoPE translation match on short prefixes, while pre-norm is slightly better on longer ones and extrapolates better beyond training lengths.Post-norm performs worse than pre-norm on long context; post-RoPE has an advantage in the memory-bound setting described in Appendix Z.Source: §3.1 · p. 5

3.2 Training the translator

  • Stage 1 initializes each map in closed form by minimizing cache-reconstruction Frobenius error, without gradient steps.The solution uses per-layer second moments and a pseudo-inverse for numerical stability.Source: §3.2 · p. 5
  • Stage 2 self-distills by minimizing divergence between the target’s predictions from its native cache and the translated cache.The training examples determine the positions where the two distributions are compared.Source: §3.2 · p. 5
  • Attention-weighted fitting emphasizes prefix positions attended to by continuation queries, and improves off-the-shelf validation loss and downstream performance over uniform Frobenius fitting.Self-distillation absorbs most of this initialization advantage, so uniform fitting remains the cheaper default; attention-weighted fitting can be used without subsequent training in compute-bound settings.Source: §3.2 · p. 6

4 Experiments

Across model pairs and cache settings, KV-Lingo generally preserves strong downstream performance, and alternating translation and generation stays close to re-prefilling across multi-turn evaluations. Translation also reduces switch time to first token substantially, with gains depending on context length and model.

4.1 Setup and baselines

  • Translators are trained per model pair on disjoint chat data and evaluated across six abilities, with closed-form linear and head-wise MLP translators among the baselines.Training prefixes range from 10 to 16k tokens; evaluation task prefixes range from a few hundred tokens to 32k.Source: §4.1 · p. 6

4.2 Shape-preserving cache translation

  • For shape-preserving Qwen3 pairs, distilled linear translators match head-wise MLP performance and reach the small model’s level in both directions.Reasoning can raise small-to-large translation scores, while closed-form translators degrade substantially when reasoning is on.Source: §4.2 · p. 7

4.3 Cross-layer cache translation

  • The assignment is nearly on par with a head-mixing MLP in both directions: 0.476 vs 0.466 in L →S and 0.577 vs 0.597 in S →L, while translating faster.Across Gemma pairs, KV-Lingo outperforms the small model in L →S for Gemma-3 with greedy match or at ν = 4, but in S →L it only reaches the small model’s level, within one standard error for Gemma-3 at ν = 4, rather than the large model’s.Source: §4.3 · p. 8

4.4 Cross-tokenizer cache translation

  • With different tokenizers, Qwen3-8B ↔ Mistral-7B-Instruct-v0.3 translation outperforms the small model on two benchmarks in L →S and four in S →L.Source: §4.4 · p. 8

4.5 MLA cache translation

  • For converted Qwen3 MLA models, KV-Lingo translates latent caches as-is, with mean scores about 0.04 below both native models across six benchmarks.KV-Lingo scores 0.588 and 0.587, compared with native-model scores of 0.629 and 0.633.Source: §4.5 · p. 9

4.6 Dense-to-MoE cache translation

  • On an Apple M3 Ultra at batch size 1, Qwen3-30B-A3B answers short-context tasks near its own-cache quality on RepLiQA and MT-Bench-101, but translation fails on long context, mostly RULER.The MoE’s 64-token prefill takes 152 ms versus 68 ms for the dense model; translation adds 1.8 ms.Source: §4.6 · p. 9

4.8 Computational gains of cache translation

  • At 32k tokens, H100 switches reach the first token 12–29× sooner; on Apple M3 Ultra, Qwen3-30B-A3B answers a 64-token prompt 9.6× sooner.Translation plus one decoding step replaces target-model re-prefilling.Source: Figure 6 · p. 10, §4.8 · p. 10

4.7 Multi-turn cache translation

  • Across CoQA, QuAC, and MultiChallenge, alternating translation and generation stays close to re-prefilling across turns, while translating the full prefix at each switch drops sharply.On QuAC, the alternating method stays within 0.05 of re-prefilling.Source: §4.7 · pp. 9–10

5 Conclusion

KV-Lingo translates caches between frozen models with token-wise linear maps trained by self-distillation, retaining strong downstream performance while avoiding target re-prefill. The surrounding analysis motivates the layer assignments and linear structure used by the method, and positions cache translation among related approaches.

5 Conclusion

  • KV-Lingo translates caches between frozen models with token-wise linear maps trained by self-distillation, retaining strong downstream performance while avoiding target re-prefill.The paper presents translation as a primitive for model hand-off and dynamic routing.Source: §5 · p. 10

A Related work

  • Prior cache-transfer approaches either require shared architecture, a target prefill, or translation mechanisms whose cost approaches re-prefilling; KV-Lingo instead fits a translator between frozen models.The related-work discussion contrasts its single-matrix-per-layer design with deeper or nonlinear alternatives and their fitting objectives.Source: A Related work · p. 17
  • Concurrent cache-transfer work confirms that a closed-form linear fit is a sound initialiser, while KV-Lingo adds behavioural distillation and measures end-to-end switching cost.The comparison also notes that some competing maps read multiple source layers, whereas KV-Lingo uses one source layer on shape-preserving pairs.Source: A Related work · p. 17
  • Related approaches to cache eviction, compression, and reuse address a single model’s cache and are orthogonal to translating between different models.The section also distinguishes cache translation from model stitching, where a map is injected once into a forward pass rather than used as the context attended over at every layer and decoding position.Source: A Related work · p. 18

B Motivating measurements: layer correspondence, linearity and rank

  • The translator must cost less than the re-prefill it replaces, motivating a single matrix per target layer applied across tokens to a selected source layer.The motivating measurements assess layer correspondence, linearity, and effective rank to evaluate this structure against cache geometry.Source: B Motivating measurements: layer… · p. 19

B.1 Layer correspondence across every pair

  • Released equal-depth Qwen3 pairs show diagonal layer correspondence, while from-scratch, Gemma-3, and unequal-depth pairs show weaker correspondence that supports selecting layers by relative depth.Qwen3 diagonal contrast is 0.30–0.34 with the per-row argmax on the diagonal; the other trained pairs retain roughly half that contrast, while random controls show little structure.Source: §B.1 · p. 19

B.2 Cache linearity and rank, per layer

Across trained model pairs, cache layers are substantially linearly related and occupy low-dimensional, anisotropic subspaces, though the fit varies by layer and position. These measurements support linear translators while identifying residual errors and depth-dependent limits.

B.2 Cache linearity and rank, per layer

  • The linear map captures 0.90–0.99 of the variance explained by an MLP, while the MLP adds at most 0.073; substantial residual variance remains, especially for values and the from-scratch pair.The residual is approximately 0.2–0.4, and the layer-wise curves show that nonlinearity is concentrated in early layers.Source: §B.2 · pp. 19, 22
  • Trained caches have effective ranks of 13–177 against d = 1024, unlike randomly initialised caches with ranks of 365–830; training concentrates cache variance into low-dimensional, anisotropic subspaces.A single principal direction carries up to half the variance in the first layers, which may reduce samples needed to estimate translators along dominant directions.Source: §B.2 · p. 19, Table 3 · p. 21
  • On the Qwen3 pairs, adding bias yields at most 0.001 R2 at any single layer, while from-scratch S →L keys gain 0.012 on average.For the Qwen3 pairs, the per-layer bias gain is an order of magnitude below the MLP gain.Source: Table 2 · p. 21
  • R2 peaks early and falls sharply at the final layer, which is the worst layer in nine of twelve pair × side × direction cells.For 8B/4B values it drops from 0.828 at layer 1 to 0.515 at layer 35; values are below keys on both Qwen3 pairs, while the from-scratch pair has early-layer exceptions.Source: §B.2 · p. 22
  • For Qwen3 8B/4B pre-normalisation keys, layer-averaged R2 stays near 0.806–0.816 across position buckets from [0, 256) to [4k, 16k), but depth slopes steepen with position.At 16k, the MLP adds 0.000 over the linear fit; long context helps the first two stack thirds and hurts the last third for values.Source: §B.2 · p. 22

C Training data and training recipe

KV-Lingo translators are trained on multi-turn prefix-and-continuation samples, with the prefix ending where a served conversation is about to receive an assistant reply. The data mixture and reasoning-mode rendering provide varied prefix lengths and preserve the inference-time placement of reasoning tokens.

C Training data and training recipe

  • Training samples are drawn from Nemotron-SFT-Instruction-Following-Chat-v2, a dataset disjoint from downstream benchmarks, and one translator per model pair is trained for generalisation across domains and languages.Samples contain a conversation prefix and its continuation, with reasoning-on and reasoning-off examples.Source: C Training data and training recipe · p. 23
  • The modified training mixture combines 50% natural samples with prefixes from 10 to 10k tokens and 50% packed samples with prefixes from 12k to 16k tokens.Continuation lengths are capped at 2048 tokens; average lengths are 750 tokens for reasoning-off and 1k for reasoning-on.Source: C Training data and training recipe · p. 23
  • Each training record uses a randomly selected assistant turn: the prefix ends at that turn’s opening header, and its answer becomes the continuation.This places the prefix at the same point where a served conversation is awaiting a response and a model switch would occur.Source: C Training data and training recipe · p. 23
  • Reasoning-on samples place the actual reasoning trace at the start of the continuation, while reasoning-off Qwen3 samples keep an empty reasoning placeholder at the end of the prefix.Earlier assistant-turn reasoning is stripped from the Qwen3 prefix; packed samples retain the same placement while moving the evaluated query to the prefix end.Source: C Training data and training recipe · p. 23
  • Stage 2 training compares the source-prefilled, translated cache with the target’s native cache on the same prefix and continuation, updating only the translator through target-model KL distillation.Source and target remain frozen; the target processes the held-back prefix token in the translated-cache path before predicting the continuation.Source: Figure 8 · p. 24

D Evaluation data

The evaluation suite spans benchmark capabilities, prefix lengths, reasoning modes, and scoring metrics, with decoding and judging protocols specified for each task. The training protocol uses a two-stage translator fit, while some benchmark pairings are omitted where floor or ceiling effects prevent informative comparisons.

D Evaluation data

  • The evaluation suite records each benchmark’s capability, translated-prefix length, scored items, reasoning mode, generation budget, and metric.MMLU-Pro and ARC-Challenge use generate-and-parse; MT-Bench-101 scores one randomly selected turn per conversation.Source: D Evaluation data · p. 25, Table 4 · p. 25
  • LongBench-v2 is omitted for specified low-performing pairs and RULER reasoning-off at 4/8/16k for Qwen3-4B↔8B, where native recall is at ceiling.The larger model is near or below the 25% chance floor in the omitted LongBench-v2 settings, while only the 32k RULER rung separates conditions for that Qwen pair.Source: D Evaluation data · p. 25

E Training protocol

  • Stage 1 solves per-layer closed-form maps from second-moment statistics collected over 400 conversations, and those maps initialise stage 2 and serve as its closed-form baseline.Stage 2 runs 5000 AdamW steps from those maps and selects the learning rate with the lowest validation KL on 64 held-out conversations.Source: E Training protocol · p. 26
  • MLP baselines replace each linear block with a two-layer MLP and train from random initialisation on cache reconstruction before the stage-2 recipe.The selected stage-2 rates are listed by pair, direction, and layer assignment, with different selection conventions for shape-preserving and cross-layer pairs.Source: E Training protocol · p. 26, Table 5 · p. 27

E.1 Seeds and error bars

The section defines how evaluation uncertainty and seed variation are reported, then shows that single-seed results can conceal substantial training noise, especially on cross-layer pairs. It also introduces the full pretrained Qwen3 evaluation suite.

E.1 Seeds and error bars

  • Error bars are standard errors over evaluation items, with aggregation adjusted for disjoint item sets, clustered questions, and multi-seed averaging.Aggregate-accuracy pairs stored only aggregate values use binomial standard errors; perplexities and validation KLs have none.Source: §E.1 · pp. 27–28
  • Seed spread matches or exceeds single-seed evaluation error on several cross-layer benchmarks, and seed 42 can depart substantially from the three-seed mean.Among 513 three-seed cells, seed 42 is more than two of its own standard errors from the mean in 54; five cross-layer cells show a collapse of at least a third on one benchmark.Source: §E.1 · p. 28
  • For shape-preserving pairs, seed spread is below single-seed evaluation error on every benchmark except XQuAD.Seed 42 falls within two standard errors of the three-seed mean on 11 of 16 reported mean-score rows.Source: §E.1 · p. 28

F Qwen3-0.6B/1.7B and 4B/8B, pretrained

  • The section evaluates pretrained 0.6B/1.7B and 4B/8B Qwen3 pairs across the suite, in both translation directions and reasoning modes.The tables report reasoning-off results, while Figures 10 and 11 show the corresponding reasoning-off and reasoning-on measurements.Source: F Qwen3-0.6B/1.7B and 4B/8B, pretrained · p. 30

G Head alignment in released Qwen3 pairs

Head-wise translators match head-mixing translators on released shape-preserving Qwen3 pairs, while from-scratch models lack the head alignment that explains this result. The closed-form map analysis finds strong same-index structure in released pairs, with a directional exception.

G Head alignment in released Qwen3 pairs

  • Released Qwen3 pairs show same-index head structure, whereas the from-scratch pair is nearly uniform; head-wise translators match head-mixing only on the released pairs.On released pairs in S →L, diagonal map norms are 33–73% above the permutation-null mean; the from-scratch diagonal is at chance, 0.125.Source: G Head alignment in released Qwen3 pairs · p. 30, Figure 14 · p. 35
  • In the 1.7B → 0.6B direction, the closed-form map’s head grids sit at chance, so they do not explain the head-wise result.This is the exception to the same-index pattern observed in both released pairs in S →L.Source: G Head alignment in released Qwen3 pairs · p. 30
  • Slicing a small model from a large one and pretraining it for 35B tokens restores same-index structure only weakly.The diagonal reaches 0.130 and 0.133, just 4% and 6% above chance.Source: G Head alignment in released Qwen3 pairs · p. 30

H Qwen3-0.6B/8B and Qwen3-1.7B/4B, pretrained

The section evaluates cross-layer Qwen3 translation, studies layer assignments and their cost-quality trade-off, and extends cache translation to Gemma pairs, different tokenizers, and MLA latents. Wider source-layer assignments improve RULER recall at every tested context budget while adding translation cost, with little LongMemEval change.

H Qwen3-0.6B/8B and Qwen3-1.7B/4B, pretrained

  • For cross-layer Qwen3, greedy match grows a target layer’s source support one layer at a time, while the R2 assignment selects the top single-layer fits separately for keys and values.The R2 assignment adds no data pass but does not account for redundancy among selected source layers.Source: H Qwen3-0.6B/8B and Qwen3-1.7B/4B,… · p. 36
  • Widening the source-layer assignment from ν = 2 to 8 increases translation cost; on cross-layer Qwen3 pairs it improves RULER recall at every context budget, but changes LongMemEval little.At 32k, translated-cache markers remain far faster than the decoding model’s own prefill; on the shortest RULER rung, one ν = 8 case is near re-prefill timing because the fixed decoding step dominates.Source: H Qwen3-0.6B/8B and Qwen3-1.7B/4B,… · pp. 36, 42

I Gemma-3-4B/12B-it

  • The Gemma-3-4B/12B-it evaluation compares relative-depth and greedy assignments, then tests wider maps using ν = 1, 2, or 4 source layers.The pair has 34 versus 48 layers and KV widths of 1024 versus 2048; its chat template has no reasoning switch.Source: I Gemma-3-4B/12B-it · p. 45, Figure 24 · p. 45, Table 14 · p. 46

J Gemma-4-E2B/E4B-it

  • The Gemma-4-E2B/E4B-it evaluation compares relative-depth and greedy within-group assignments in both directions, as closed-form fits and after self-distillation.MT-Bench-101 and LongBench-v2 use reasoning-on; the other listed benchmarks use reasoning-off.Source: J Gemma-4-E2B/E4B-it · p. 48, Figure 25 · p. 48, Table 16 · p. 49

K Cross-tokenizer cache translation: Qwen3-8B/Mistral-7B-Instruct-v0.3

  • Across the Qwen3–Mistral tokenizer boundary, source tokens are assigned across target tokens within minimal groups covering matching character spans, and translated keys use assigned target positions for rotation.On the fitting corpus, 83% of minimal groups contain one token on each side; stage 1 regresses each source token’s keys and values onto the group’s average target keys and values.Source: K Cross-tokenizer cache translation:… · pp. 50–51

L MLA cache translation

  • For the MLA pair, the source and target cache each hold a 1024-dimensional compressed latent and a 64-dimensional rotary key per token and layer.The models were converted with CARE and recovery-fine-tuned; the section compares translating latents against translating keys and values.Source: L MLA cache translation · p. 53, Table 18 · p. 54, Table 19 · p. 54

M Comparison with DroidSpeak

The comparison finds that DroidSpeak’s verbatim cache reuse fails for this cross-size Qwen3 pair unless much of the target stack is recomputed. KV-Lingo uses less target-side compute and can transfer some source-model advantage, while DroidSpeak ships a smaller cache subset.

M Comparison with DroidSpeak

  • KV-Lingo costs 3% of the target prefill in S →L but loses accuracy, while DroidSpeak matches target accuracy at 93% of prefill; in L →S, DroidSpeak recomputes the whole stack.KV-Lingo outscores the 0.6B target on ARC-C, 0.616 vs. 0.568.Source: M Comparison with DroidSpeak · pp. 55–56, Table 20 · p. 55
  • Verbatim reuse stays near zero and below chance through j = 18; the target’s own accuracy is first recovered at j = 22, after recomputing 79% of its layers.DroidSpeak cannot exceed target accuracy because it recomputes all layers it does not reuse.Source: M Comparison with DroidSpeak · p. 55, Table 21 · p. 55
  • DroidSpeak ships only the two reused layers, versus all 28 layers for a translator, but its design premise of nearly interchangeable caches does not hold across these model sizes.The tested pair has matching cache shape but different hidden widths, requiring recomputation to begin at layer 0.Source: M Comparison with DroidSpeak · pp. 55–56

N Dense-to-MoE cache translation

The Qwen3-4B/Qwen3-30B-A3B study evaluates translation in both directions across reasoning modes and task groups, then compares linear translators with larger MLP variants and measures switch latency.

N Dense-to-MoE cache translation

  • Each target layer maps keys and values from three source layers; the 4B-to-MoE translator has 151M parameters, and the reverse translator has 113M.Both models share a tokenizer, while the MoE has 48 layers and the 4B has 36.Source: N Dense-to-MoE cache translation · p. 57
  • The MoE-to-4B translator falls short of the 4B’s native cache in 32 of 36 evaluation cells, and its RULER recall is 0.07–0.28.The suite covers both reasoning modes; Figure 29 groups results by short judged, short other, and long-context tasks.Source: N Dense-to-MoE cache translation · p. 57, Table 22 · p. 57, Figure 29 · p. 58
  • The 4B-to-MoE MLP uses 10× as many parameters as the linear translator, ends at held-out KL 0.0644 versus 0.0634, and takes 1.8× as long to train.In the reverse direction, the MLP is 8× larger and ends at 0.0842 versus 0.0858, while taking 1.5× as long to train.Source: N Dense-to-MoE cache translation · p. 58, Table 23 · p. 58
  • On an Apple M3 Ultra with MLX in bf16 at batch size 1, translating the 4B cache plus one MoE decoding step yields first-token times of 15.8 ms (9.6× sooner) at 64 tokens and 31.7 ms (13.3× sooner) at 1024, versus the MoE’s own prefill.Source: N Dense-to-MoE cache translation · p. 59, Table 24 · p. 59

O Measuring the distance between caches

This block examines how cache-fitting objectives and capture points relate to downstream decoding, how reasoning changes translated-cache scores, and how caches behave across repeated switches and altered handoff points.

O Measuring the distance between caches

  • A better cache fit does not ensure better decoding: post-k-norm has higher R2 but worse downstream performance, while attention-weighted fitting improves long-context results over Frobenius fitting.For the long-context recipe, attention weighting changes RULER recall from 0.10–0.35 to 0.60–0.84; the static coordinate reweightings collapse to the Frobenius solution, so only position-varying weights change the fit.Source: O Measuring the distance between caches · pp. 60–61, Table 25 · p. 60
  • Attention-weighted initialization reaches in 250 steps the validation KL that Frobenius initialization needs 5000 steps for, though stage 2 absorbs most of the downstream gap.After stage 2, the two are within 0.03 recall at every RULER length; attention weighting costs an extra attention pass during fitting and nothing at inference.Source: O Measuring the distance between caches · p. 61, Figure 30 · p. 61

P The impact of reasoning on cache translation

  • On the 0.6B/1.7B pair, translated-cache reasoning boosts are +0.102 versus +0.018 when translating into the larger model, and +0.029 versus +0.045 when translating into the smaller one.For the 4B/8B pair, translated caches track the target’s native reasoning boost in both directions.Source: P The impact of reasoning on cache… · p. 62, Figure 31 · p. 62

Q Multi-turn cache translation

  • KV-Lingo retains each model’s own and translated cache spans, translating only newly written spans at each switch; it stays with full re-prefill through turn 8 on QuAC and the 9–10 bin on MultiChallenge.The naive strategy falls below full re-prefill from turn 3 on QuAC and in most MultiChallenge bins; only the active cache line needs to remain on the accelerator.Source: Q Multi-turn cache translation · p. 63, Figure 33 · p. 64, Figure 34 · p. 64

R Pre-k-norm vs. post-k-norm translation

S Pre-k-norm versus post-RoPE translation

T Moving the switch point

  • Translating only the document while the target prefills the question keeps self-distilled translators close to whole-prompt results, with largest drops of 0.07 on LongBench-v2 in L →S and 0.05 on RULER at 32k in S →L.After the source’s reasoning trace is handed over, the 0.6B target matches the 1.7B source on MMLU-Pro and ARC-C and exceeds it on GSM8K.Source: T Moving the switch point · p. 68, Table 29 · p. 68, Table 30 · p. 69
  • LongMemEval absolute scores in Table 29 are not comparable with Table 8 because Table 29 uses 256 questions per rung, but the paired difference between protocols is comparable.In collapsed closed-form cells, up to 0.96 of items have no parseable answer; trained maps answer every item in every cell.Source: T Moving the switch point · p. 68

U Objective ablation: cross-entropy and mixed objectives

The objective ablation compares self-distillation with ground-truth cross-entropy and a mixed objective, finding cross-entropy less robust while the mixed objective performs similarly to self-distillation.

U Objective ablation: cross-entropy and mixed objectives

  • Cross-entropy leaves a 0.048–0.138-nat gap to native decoding, versus 0.031–0.101 for self-distillation, and can destabilize long-context or reasoning-on decoding.Reported failures include unclosed reasoning blocks, repetition loops, and collapsed RULER recall and MT-Bench scores.Source: U Objective ablation: cross-entropy… · p. 70
  • Training on target-generated continuations might mitigate the reported cross-entropy brittleness; this study instead uses ground-truth continuations, typically generated by more capable models.The text presents target-generated continuations as a possible mitigation, not as an evaluated result.Source: U Objective ablation: cross-entropy… · p. 70
  • The mixed objective performs about as well as self-distillation across the reported evaluations, with most differences within one standard error.The mixed loss adds a cross-entropy term to self-distillation with λ = 1.Source: U Objective ablation: cross-entropy… · p. 70

V Cache translation with Qwen3-like models pretrained from scratch

On Qwen3-like pairs pretrained from scratch, self-distillation makes cache translation nearly as effective as on released pairs, while head mixing and split key/value maps remain effective design choices. English-only-trained translators also generalize across languages.

V Cache translation with Qwen3-like models pretrained from scratch

  • The from-scratch Qwen3-0.6B/1.7B pair uses 35B tokens per model, independent initialisations, and no distillation; self-distillation closes most of its closed-form translator’s larger gap.The closed-form gap is 1.8–4.7× the released pair’s, though the authors note this may reflect the recipe or smaller training budget.Source: V Cache translation with Qwen3-like… · p. 71
  • Self-distillation cuts the from-scratch pair’s log-perplexity gap by 5–9× to 0.031, 0.067, 0.043, and 0.101 nats, leaving 0.017–0.037 nats between pairs.These are the four L→S and S→L results on Nemotron-CC and WikiText; neither pair is clearly easier to translate after self-distillation.Source: V Cache translation with Qwen3-like… · p. 71, Figure 36 · p. 72
  • After self-distillation, the from-scratch pair’s average-accuracy costs are 0.006 and 0.001, versus −0.020 and 0.026 for the released pair.The negative value means the translated cache beats the target’s own cache.Source: V Cache translation with Qwen3-like… · p. 71, Figure 36 · p. 72
  • On independently initialized heads, KV-Lingo wins all twelve matched-budget and 4×-budget rows; the head-wise MLP leaves 1.7–3.4× its perplexity gap at matched budget.On released Qwen3 pairs with aligned heads, head-wise translators instead match head-mixing ones.Source: V Cache translation with Qwen3-like… · p. 71, Table 31 · p. 72, Table 32 · p. 73
  • For released Base pairs, S→L translated-cache accuracy across four tasks lands within 0.005 of the small model, not near the large target.This result concerns the released Base pairs, not the Instruct checkpoints used in the main results.Source: V Cache translation with Qwen3-like… · p. 73, Table 33 · p. 74
  • Off the fitting distribution, the Base-pair translators’ median relative perplexity gap is 2.1% and the worst is 6.5% across twenty cells.The four evaluation corpora were never seen during fitting.Source: Table 34 · p. 74

W Split vs. joint maps

  • The split key/value map loses little to a joint map: validation KL rises 0.0008–0.0027 nats per token, continuation log-perplexity at most 0.008 nats, and BoolQ accuracy at most 0.013.The BoolQ differences are within sampling error; the split map halves the joint map’s parameters.Source: W Split vs. joint maps · p. 75, Table 35 · p. 75

X Cache translation across languages

Y Computational gains of cache translation

KV-Lingo replaces target-model re-prefill with cache translation and one decoding step, reducing switch time while retaining native decoding throughput. A separate memory-budget path translates attention operations without materializing the target cache, but its head-wise setting has a performance trade-off.

Y Computational gains of cache translation

  • Translation applies per-token key and value maps without attention, then the target performs one decoding step; re-prefill instead runs the target over the full prefix.On shape-preserving Qwen3 pairs, translation costs 117 MFLOP per token into 28-layer models and 151 MFLOP into 36-layer models.Source: Y Computational gains of cache… · p. 77
  • After a switch, the 0.6B/1.7B pair decodes at the same 34 tokens/s with either cache, at every tested length.The translated cache has the target model’s own shape.Source: Y Computational gains of cache… · p. 77
  • At 32k on the 0.6B/1.7B pair, KV-Lingo peaks at 8.7 GB, versus 9.5 GB into 0.6B and 10.3 GB into 1.7B for re-prefill.Translation skips the prefill’s activations.Source: Y Computational gains of cache… · p. 77
  • At 32k on H100, TTFT is 12.0×, 14.8×, 23.2×, and 28.9× faster into Qwen3-0.6B, 1.7B, 4B, and 8B; Apple M3 Ultra reaches 9.6× at 64 tokens.The Apple result is for switching from Qwen3-4B to Qwen3-30B-A3B.Source: Y Computational gains of cache… · p. 77, Table 37 · p. 78

Z Decoding under a memory budget without materialising the target cache

  • Under a memory budget, a linear post-RoPE map can move across attention, applying to pooled values so the map runs once per layer and decoding step rather than per cached token.The context remains stored as the small model’s cache; the target decodes directly from it.Source: Z Decoding under a memory budget… · p. 79
  • Throughput acceleration in the memory-budget setting requires head-wise maps, but these score below the large model when cache shapes differ; Table 38 is a random-weight proof of concept.A head-mixing map makes every target head pay to score against all source heads at every cached token.Source: Z Decoding under a memory budget… · p. 79, Table 38 · p. 79

Related papers

Loading 2609.32610v1…