Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration

1 INTRODUCTION

• KV-cache quantization compresses individual entries but retains them all, whereas linear attention aggregates historical contributions into a fixed-size state that can cause interference.The contrast motivates bridging per-KV compression and multi-KV aggregation within one efficient-attention mechanism.Source: §1 · p. 1
• In a restricted RAM-Net construction, soft address assignments extend hard quantized matching, and separable read-write overlap locally approximates full-attention similarity.The overlap supports recurrent aggregation of historical KV contributions into continuous slot states.Source: §1 · pp. 1–2
• Transformer-to-RAM-Net migration uses a soft-quantized intermediate model, then initializes RAM-Net from its parameters and progressively fine-tunes it.The intermediate model aligns query and key projections, folds in a codebook, and copies remaining shared parameters from the Transformer.Source: §1 · p. 2
• 87.1% of teachers’ accuracy gains over random guessing were recovered across six commonsense and knowledge tasks with 500M tokens per model.The evaluation covered nine pretrained Transformers from five model families, with MHA and GQA configurations and sizes from 0.3B to 7B parameters.Source: §1 · p. 2

2 BACKGROUND

• Full attention predicts a value as a softmax-weighted sum over historical values, using query-key dot-product similarities that act as soft nearest-neighbor matching.Because it uses all historical KV samples, storage and per-query computation grow with context length.Source: §2 · p. 2

2.1 QUANTIZED KV CACHES: DISCRETE-CODE SOFT-NN PREDICTOR

• KV-cache quantization encodes historical KV samples as finite numerical levels or codewords, reducing per-sample storage while retaining all samples.Matching quality depends on whether reconstructed keys preserve query-key similarities.Source: §2.1 · p. 2

2.2 LINEAR ATTENTION: LINEAR PREDICTOR WITH CONTINUOUS FEATURES

• Linear attention replaces exponential dot-product similarity with separable query-key features, enabling recurrent aggregation into a fixed-size continuous state.The fixed state makes storage and per-query computation independent of context length, but limits fine-grained matching to historical keys.Source: §2.2 · p. 3

2.3 RAM-NET: DISCRETE-ADDRESS LINEAR-INTERPOLATION PREDICTOR

• RAM-Net maps keys and queries to sparse write and read distributions over discrete addresses, whose slots store recurrently updated continuous states.Product Softmax forms the distributions and Top-K retains selected probabilities; GSU updates selected slots with values.Source: §2.3 · p. 3
• Read-write address overlap defines RAM-Net’s output, combining quantized-cache discrete codes with linear attention’s recurrent continuous-state aggregation.The overview figure presents this bridge through circular soft addressing and a separable overlap that locally approximates Soft-NN matching.Source: §2.3 · p. 3, Figure 2 · p. 4

3.1 DISCRETE-CODE REPRESENTATION VIA CIRCULAR SOFT ADDRESSING

• The address space is a product of U factors, with each address indexing a slot and each factor selecting one of dp codes.Read and write projections produce factor-wise logits, which Product Softmax combines into a distribution over addresses.Source: §3.1 · p. 4
• As assignment concentration β→∞, Product Softmax selects each factor’s maximizing code, recovering hard code assignment in the fixed-codebook, rank-at-most-two construction.The restricted construction uses a shared fixed codebook and rank-at-most-two read and write projection blocks.Source: §3.1 · pp. 4–5

3.2 APPROXIMATION TO SOFT-NN PREDICTORS VIA ADDRESS OVERLAP

• In the local regime, read-write address overlap approximates exponential dot-product similarity and separates exactly into read and write terms, avoiding key reconstruction.The approximation and separability establish links to continuous Soft-NN, discrete-code Soft-NN, and linear predictors.Source: §3.2 · p. 5

3.3 LINEAR PREDICTOR VIA SLOT AGGREGATION

• Separable address features aggregate historical KV contributions by slot; GSU adds controlled retention, while the cumulative case reduces to averaging when forgetting is removed.The readout uses each slot’s read weight and accumulated write mass.Source: §3.3 · pp. 5–6
• Mass-aware readout generally matches Transformer attention more closely before head-level recovery but tends to yield slightly higher perplexity after migration than the original readout.The authors note a possible mismatch between mass reweighting and Top-K selection based only on read scores.Source: §3.3 · p. 6

4 TRANSFORMER-TO-RAM-NET WEIGHT MIGRATION

• Migration has two stages: preserve teacher token-to-token attention patterns in a soft-quantized intermediate model, then enable RAM-Net recurrent updates and progressively fine-tune to reduce output discrepancies.The first stage adapts query-key matching; the second recovers outputs after conversion to RAM-Net.Source: §4 · p. 6

4.1 MIGRATION TO SOFT-QUANTIZED INTERMEDIATE ATTENTION

• Circular read-write address overlap replaces exponential dot-product similarity to align token matching before GSU, with aligned query and key projections folded into read and write projections.The intermediate model is constructed through low-rank and positional alignment of the teacher’s query and key projections.Source: §4.1 · p. 6
• Adapters compress each head’s query and key vectors to 2U dimensions; RoPE and NoPE candidates are refined against teacher attention maps, with head assignments based on relative alignment errors.The migration retains RAM-Net’s original CAPE pattern across layers.Source: §4.1 · p. 6

4.2 RAM-NET CONVERSION AND PROGRESSIVE RECOVERY

• Recovery converts the intermediate model to RAM-Net, matches head outputs and attention-branch residual updates, then fine-tunes all student parameters using next-token cross-entropy.Head and attention-block recovery use normalized mean squared error on their respective outputs.Source: §4.2 · p. 7

5.1 EXPERIMENTAL SETUP

• The nine public teacher models span 340M to 7B parameters and use a shared 500M-token migration budget per model.Models match their teacher’s architecture and tokenizer apart from the attention mechanism; the tokens are sampled from FineWeb-Edu.Source: §5.1 · p. 7

5.2 LANGUAGE MODEL EVALUATION BENCHMARKS

• 87.1% is the mean relative score across teachers, with scores ranging from 79.2% to 95.0%, indicating retention of most teacher performance across attention types, families, and scales.The evaluation covers six commonsense and knowledge tasks; Table 1 reports language-modeling and task accuracy.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 ABLATION STUDIES

• The circular codebook yields lower attention JS than all tested alternatives across the nine teachers.Alternatives include shared random codes, orthogonal random codes, shared k-means, and per-factor product quantization.Source: §5.3 · p. 8, Table 2 · p. 9
• β⋆ gives near-minimal normalized attention JS across teachers, although some minima occur around 1.58β⋆.The sweep holds low-rank adapters constant and normalizes JS by each teacher’s value at β⋆.Source: §5.3 · p. 8, Figure 3 · p. 8
• Under a fixed 500M-token budget, the complete pipeline improves Wiki PPL and downstream accuracy over full-model-only recovery; removing either recovery stage lowers downstream accuracy.Pattern-free CAPE selection slightly lowers logit KL and Wiki PPL but also lowers downstream accuracy relative to the preset pattern.Source: §5.3 · p. 8

6 RELATED WORK

• Table 4 presents pipeline ablations on Qwen2.5-1.5B.The table caption identifies the model and the scope as pipeline ablations.Source: Table 4 · p. 9

6.1 LINEAR-TIME ATTENTION VIA QUANTIZATION AND KERNEL APPROXIMATION

• RAM-Net uses discrete-code soft assignments whose separable read-write overlap locally approximates exponential dot-product similarity and supports recurrent aggregation.The discussion connects this construction to vector-quantized attention and kernelized attention approaches including FAVOR+.Source: §6.1 · p. 9

6.2 TRANSFORMER-TO-LINEAR WEIGHT MIGRATION

• Unlike methods that generally convert Transformers directly to linear or recurrent targets, RAM-Net proposes an intermediate construction derived from quantized Transformers.Attention-to-Mamba also uses an intermediate, but it is kernelized linear attention (Moudgil et al., 2026).Source: §6.2 · p. 9

7 CONCLUSION AND LIMITATIONS

• The paper establishes RAM-Net as a theoretical bridge between discrete-code quantization and linear attention and uses the connection for Transformer-to-RAM-Net weight migration.Source: §7 · p. 9
• The derivation is local and assumes a fixed circular codebook with projection blocks of rank at most two, unlike practical RAM-Net’s unrestricted projections.Source: §7 · p. 9
• Migration experiments use a fixed token budget without further tuning.Source: §7 · p. 9

A.1 FULL PROOF AND EXTENSIONS OF THEOREM 1

• Theorem 2 generalizes the proof to separate read and write assignment concentrations and derives an explicit approximation error bound.The continuous-codebook dot-product approximation is extended to separate read and write concentrations.Source: §A.1 · pp. 13–14
• For fixed U and dp ≥4, the combined continuous- and finite-codebook error is O(UM^4).Source: §A.1 · p. 16

A.2 FROM SOFTMAX ATTENTION TO MASS-AWARE RECURRENT READOUT

• The mass-aware readout weights each address by its read probability and accumulated write mass, using the write-weighted mean of values stored there.Source: §A.2 · pp. 16–17

A.3 GSU DYNAMICS AND THE CUMULATIVE-AVERAGING LIMIT

• In the stated GSU limit, the state becomes the cumulative write-weighted mean, and with mass-aware readout it exactly reproduces normalized address-kernel attention.Source: §A.3 · p. 17

A.4 EFFECTS OF CODEBOOK SIZE, SPARSITY, AND READOUT

• At moderate β, increasing dp refines the angular grid and smooths value-field transitions toward full attention; small-β Top-K causes larger artifacts.Mass-aware readout changes slot influence when address masses differ and can produce oblique transitions.Source: §A.4 · p. 17, Figure 4 · p. 18

A.5 HARD-WRITE LIMIT AND VECTOR-QUANTIZED ATTENTION

• As βw→∞ with finite βr, writes become nearest-code assignments while reads stay soft, yielding softmax attention over product-quantized keys.After the cumulative-averaging limit, mass-aware readout recovers the per-code aggregation underlying Transformer-VQ.Source: §A.5 · pp. 18–19

A.6 CORRESPONDENCE BETWEEN ROPE AND CAPE

• RoPE translates continuous waves before sampling and holding, whereas CAPE shifts sampled-and-held logits; this operation-order difference introduces discretization error in address distributions.Source: §A.6 · pp. 19–20

A.7 LOW-RANK ADAPTER CONSTRUCTION

• The adapter initialization uses leading singular components for rank-one coefficient approximations, but separate truncations do not generally minimize the joint constrained objective.They provide a structured initialization refined by forward KL alignment; the NoPE construction instead uses a truncated SVD of the position-averaged teacher operator.Source: §A.7 · p. 21

B.1 TEACHER MODELS

• The evaluation covers nine pretrained teachers from five families, with different head dimensions, query/KV-head counts, and attention configurations.Source: §B.1 · p. 22

B.2 MIGRATION DETAILS

• Migration analytically folds aligned adapters into RAM-Net projections without training during circular soft quantization, then trains routing and forgetting parameters in head-level recovery.The experiments use a 500M-token budget per teacher; copied V/O projections and the backbone remain frozen during head-level recovery.Source: Table 6 · p. 22, §B.2 · p. 23

• Mass-aware readout yields lower attention JS in six of nine models before head-level recovery, but generally higher Wiki PPL with comparable common-knowledge performance after migration.Source: §B.3 · p. 23, Table 8 · p. 24
• The post-migration gap may relate to coupling read weights to forgetting dynamics and a potential mismatch between mass reweighting and Top-K selection based only on read scores.Source: §B.3 · pp. 23–24
• In 340M train-from-scratch results, mass-aware readout raises average common-knowledge score from 55.6 to 56.8 while giving slightly higher Wiki PPL.Source: §B.3 · p. 24, Table 9 · p. 24

Head NMSE with concentration β

• Head NMSE is minimized near β⋆ across teachers, with some minima around 1.58β⋆, supporting β⋆ as a balanced default setting.This trend is consistent with the attention-JS results.Source: §B.4 · p. 24

Different dp and U

• Although increasing U generally reduces intermediate forward KL and head NMSE, U = 5 achieves the best final Avg. and relative score despite U = 6 having lower low-rank KL.The chosen balance is dp = 4 and U = 5; larger address spaces can reduce overlap mass and effective slot utilization.Source: §B.4 · pp. 24–25, Figure 6 · p. 25

LongBench

• On LongBench, RAM-Net recovers teacher performance more closely for smaller models, while the gap generally widens at higher scales under the shared 500M-token budget.The results suggest larger teachers may require a larger recovery budget to better transfer their downstream capabilities.Source: §B.4 · p. 24, Table 10 · p. 25

Staged migration

• Under the same 500M-token budget, the complete pipeline overtakes full-model-only recovery after attention-block recovery and finishes with higher Avg. and relative score R.Omitting either head-level or attention-block recovery lowers final Avg. and R, even when Wiki PPL remains similar.Source: §B.4 · p. 25, Table 12 · p. 26

Head-recovery loss curves

• The circular codebook starts with the lowest head NMSE and reduces error fastest during recovery, improving both initial attention alignment and subsequent recovery efficiency.The comparison extends the low-rank alignment and circular-codebook results to head-level recovery.Source: §B.4 · p. 24, Figure 7 · p. 26

Per-stage breakdown

• In FLA-2.7B, Top-K truncation causes the largest attention-KL increase, yet head-level recovery rapidly reduces output NMSE while attention KL remains high.Attention-block and full-model recovery further reduce output discrepancy; full-model recovery slightly increases NMSE as objectives shift to final-logit KL and cross-entropy.Source: §B.4 · p. 24, Figure 8 · p. 27