PaperlayerView exact revision on arXiv

Source-linked AI summary

Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs

Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng

arXiv:2610.35985v1cs.LGcs.AI

TL;DR

Conditional-memory tables store long-tail facts, but low-bit quantization and frequency-based precision allocation can leave important lookups under-protected. GateRQ uses a 1-bit code to estimate each lookup’s gate and selectively reads deeper residual codes. Across two models and three codecs it outperforms alternative selection rules, while approaching FP8 PopQA accuracy at a lower read budget.

  • Problem

    Conditional-memory tables hold long-tail facts, but quantization erases them first and frequency-based precision allocation can prioritize low-gate lookups.

    Source: §1 · pp. 1–2

  • Method

    GateRQ stores rows as ladders of 1-bit residual codes and uses the gate computed from level 1 to select deeper reads per lookup.

    Source: §1 · p. 2

  • Results

    Across two models and three codecs, GateRQ beats other selection rules in all 36 paired PopQA comparisons at equal read bits and, with TCQ, is within 0.2 PopQA points of FP8 at 2.10 bits.

    Source: §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    On DeepSeek-V4.1-Flash, GateRQ cuts table memory from 23.6 to 3.0 GiB per GPU and raises vLLM throughput by 11%–31% over FP8 tables in GPU memory.

    Source: §1 · p. 2

    The distortion analysis assumes lookup errors are uncorrelated and isotropic.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Conditional memory layers, such as DeepSeek's Engram, attach large hashed n-gram tables to a language model, and removing them from DeepSeek-V4.1-Flash costs 36.8 PopQA points. Low-bit quantization loses long-tail facts first, and the standard remedy, giving more bits to frequently accessed rows, backfires: refining lookups by access frequency is worse than refining them at random. We trace this to the gate that scales each retrieved row. A lookup's quantization error is weighted by its gate, which is small for frequent n-grams, large for the rare n-grams of long-tail entities, and computable from a 1-bit code. GateRQ therefore stores each row as a ladder of 1-bit residual codes, keeps level 1 in GPU memory to compute the gate, and reads deeper levels only for the lookups the gate ranks highest. Without retraining, GateRQ cuts GPU table memory by 7.8x on DeepSeek-V4.1-Flash, which raises vLLM throughput by 11%-31% over FP8 tables, while losing about 1 PopQA point at 1.50 bits read per lookup and 0.2 points at 2.10 bits across two models.

1 Introduction

Conditional-memory tables hold long-tail facts but are large, and low-bit quantization threatens rare-entity knowledge. GateRQ allocates precision per lookup using its gate, rather than allocating it by row frequency.

1 Introduction

  • Conditional-memory tables hold much of the model’s factual knowledge, and zeroing them costs DeepSeek-V4.1-Flash 36.8 PopQA points.The tables occupy 203 GB (189 GiB) in FP8, or 40% of the checkpoint.Source: §1 · p. 1
  • Quantization erases long-tail facts first, making equal precision across rows a poor fit for conditional memory.Frequency-based allocation is commonly used for recommender-system embedding tables, but assumes each access contributes equally to output error.Source: §1 · p. 1
  • Frequency-based precision allocation fails because frequent n-grams receive small gates and rare named-entity n-grams receive large gates.Rows seen more than a thousand times account for a third of block 2’s 2-gram lookups but under 4% of output error under uniform quantization.Source: §1 · p. 2
  • Expected-gate row ranking still misses important lookups because rare-row gates are poorly estimated and common-row gates vary with context.Gates computed from a 1-bit code closely match FP8 gates (r ≥0.989), allowing lookup precision to be chosen before high-precision data is read.Source: §1 · p. 2
  • GateRQ stores each row as a ladder of 1-bit residual codes and reads deeper levels only for lookups whose gates rank them as important.Level 1 stays in GPU memory to obtain the lookup key and gate; deeper levels refine the value under a fixed bits-per-lookup budget.Source: §1 · p. 2
  • At equal bits per lookup, GateRQ beats three selection baselines in all 36 paired PopQA comparisons; TCQ at 1.50 and 2.10 bits costs about 1 and 0.2 PopQA points versus FP8.In vLLM, GateRQ also cuts table memory from 23.6 to 3.0 GiB per GPU and raises throughput by 11%–31% over FP8 tables in GPU memory.Source: §1 · p. 2

2 Background: Conditional Memory Layers

Conditional memory retrieves rows from hashed n-gram tables and uses context-dependent gates to control their contributions. The two studied models implement this mechanism with different table and convolution configurations.

2 Background: Conditional Memory Layers

  • Engram maps canonicalized suffix n-grams to rows through multiple hash heads, which address tables to manage the large n-gram space.A row access is defined by token position and head; its row ID depends on tokens, not the hidden state.Source: §2 · p. 2
  • The hidden state queries retrieved memory to compute a scalar gate, which suppresses contributions when memory and context disagree.Engram visualizations show gates opening for local patterns such as multi-token named entities and fixed phrases [1, Figure 7].Source: §2 · p. 3
  • In the original Engram layer, gated values pass through a short depthwise causal convolution before the output is added to the backbone.The retrieved row affects the layer through two paths: its key sets the gate, while its value supplies content to the residual stream.Source: §2 · p. 3
  • DeepSeek-V4.1-Flash uses Engram layers in blocks 2 and 15, while Qwen3.8-Flash-Next uses one PLE layer in block 2.DeepSeek removes the short convolution; Qwen retains a width-4, dilation-3 convolution.Source: §2 · p. 3

3 Where Quantization Loss Comes From

This section examines where quantization loss arises in gated conditional memory and whether important lookups can be identified before reading higher-precision data. It finds that lookup importance depends on the gate, not access frequency, and varies with context.

3.1 Distortion of a Lookup

  • Equation (5) models layer distortion as a sum over lookups, with each lookup's importance combining gate energy, head gain, and row energy.The authors treat lookup errors as uncorrelated and isotropic; they describe Equation (5) as a local proxy rather than task loss.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2 Three Observations

  • Gate error is negligible: gates from the 1-bit table correlate 0.989–1.000 with FP8 gates, while quantizing only values costs as much as quantizing both.The top 10% of lookups by importance overlap by 98.1% in block 2 and 92.8% in block 15; gates-only accuracy costs at most 0.22 PopQA points at 95% confidence.Source: §3.2 · p. 4
  • Frequent rows serve 32% of block 2's 2-gram lookups but carry only 3.8% of importance; rows unseen in the counting corpus serve 14% and carry 23%.Giving extra precision to the 41% most-frequent lookups removes 26% of the error that refining all lookups would remove, versus 41% for a random 41%.Source: Figure 1 · p. 4, §3.2 · p. 5
  • The top 10% of lookups carry 48% of importance in block 2 and 69% in block 15 on held-out English text, but context and rare rows defeat row-level estimates.On RACE passages, the top 10% carry 50% and 47%, respectively; per-lookup ranking by the current gate beats ranking by expected row gate in every tested setting.Source: §3.2 · p. 5

4 GateRQ

GateRQ combines a residual-code ladder with per-lookup depth allocation: all lookups obtain a level-1 gate, and only selected lookups read deeper levels. Its allocation rule uses estimated importance and a fixed read budget to decide which lookups receive additional precision.

4 GateRQ

  • GateRQ pairs a ladder that raises precision after a row is read with a rule that sets each lookup’s depth from its gate.Level 1 supplies the gate; deeper levels refine only the value.Source: §4 · p. 5, §4.1 · p. 5

4.1 Residual Ladder

  • Rows use up to three 1-bit residual levels, with level 1 on GPU and deeper levels on host; reconstruction error stays close to direct 2- and 3-bit codes.Each level encodes the residual left by preceding levels, and lookups read only the levels they need.Source: §4.1 · pp. 5–6

4.2 Per-Lookup Depth Allocation

  • A lookup is refined from level k - 1 to k when its importance exceeds the bit-price threshold; decreasing marginal error reductions make selected depths nested.This applies the classical marginal-return rule of bit allocation to lookups rather than table rows.Source: §4.2 · p. 6
  • The allocator estimates importance from level 1 and uses per-batch quantile thresholds so fixed fractions of lookups read at least levels 2 and 3.The top s2 fraction reads at least level 2, and the top s3 fraction reads level 3; fractions are fixed in advance to give every batch the same read rate.Source: §4.2 · pp. 6–7

5 Experiments

Across two conditional-memory models and three codecs, GateRQ leads the other lookup-selection rules at matched read budgets, while its TCQ results remain close to FP8. Ablations favor gate-based selection, and efficiency measurements show reduced table loading and table-side time.

5.1 Setup

  • The evaluation covers two models and four benchmarks, comparing FP8 and uniform residual reconstruction with three selection rules across TCQ, INT1, and PQ1 ladders.The selection rules use the same refinement fractions and read budgets as GateRQ.Source: §5.1 · p. 7

5.2 Main results

  • GateRQ has the best PopQA accuracy in all 12 model–codec–budget settings; with TCQ it is within 0.2 PopQA points of FP8 at 2.10 bits, a nonsignificant difference.Its four-task mean is within 0.3 points of FP8 at both budgets; these comparisons do not establish equivalence.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Ablation

  • At the 1.50-bit setting, gate ranking removes 93% of the error removable by refining every lookup, versus 70% for Prior, 41% for Random, and 26% for Frequency.Measured PopQA recovery follows the same order: 96%, 58%, 41%, and 34% of the gap between level 1 and two full levels.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4 Efficiency

  • At 8,192 tokens, GateRQ loads 2.25–3.15 MiB per GPU and takes 1.3–1.6 ms on the table side, versus 12.4 MiB and 3.1 ms for GPU-resident FP8 tables.The table side is about 1% of prefill, so these timing differences do not appear in end-to-end throughput.Source: §5.4 · pp. 9–10, Figure 5 · p. 9

6 Conclusion

GateRQ compresses conditional-memory tables after training and assigns precision per lookup using the model’s gate. The paper positions this approach alongside trained table designs, storage-hierarchy work, and input-level routing as a complementary way to reduce table storage and transfer.

A Related Work

  • Related work includes trained table parameterizations [19]–[22], storage-hierarchy approaches [23] [24], and routing whole inputs or layers [25].GateRQ is complementary: it compresses existing checkpoints and makes per-lookup decisions rather than input-level routing.Source: A Related Work · p. 12

6 Conclusion

  • GateRQ compresses existing checkpoints after training without gradient updates, setting each lookup’s precision from the model’s gate.The conclusion reports that it beats other selection rules across 36 paired PopQA comparisons and, with TCQ at 2.10 bits, comes within 0.2 PopQA points of FP8.Source: §6 · p. 10, A Related Work · p. 12

B Models and Tables

The conditional-memory tables hold substantial model knowledge, while the supporting analyses examine how popularity and projection structure relate to quantization error. Loss from using only 1-bit level 1 varies by subject popularity, and measured row and projection properties support the assumptions used to analyze that error.

B Models and Tables

  • Zeroing the tables costs DeepSeek 36.76 PopQA points and Qwen 12.90 points; the tables occupy 202.8 GB and 51.2 GB in FP8, respectively.The losses are about 70% and 33% of each model’s accuracy; the table storage is 39.7% and 27.6% of their respective checkpoints.Source: B Models and Tables · p. 12, Table 4 · p. 12

C Supporting Analyses for Section 3

  • The supporting records cover held-out English, Chinese, code, RACE, and PopQA prompts, including 1,279,844 PopQA prompt positions.The PopQA set contains 4,000 questions, each with the question and 15 few-shot examples in its prompt.Source: C Supporting Analyses for Section 3 · p. 13

C.1 Loss by Subject Popularity

  • Using only level 1 loses 6.24 PopQA points on DeepSeek’s rarest subject quartile versus 1.51 on its most popular; Qwen’s largest loss is 3.67 points on its second quartile.On Qwen, that second quartile also has the lowest FP8 accuracy (26.85%) and exceeds the rarest-quartile loss of 2.95 points; both models lose least on the most popular quartile.Source: §C.1 · p. 13, Table 5 · p. 13, Figure 6 · p. 13

C.2 Assumptions of Equation (5)

  • DeepSeek table rows are close to independent Gaussian vectors: their largest-absolute-value-to-RMS ratio has median 3.005, and dimension standard deviations differ by at most 5%.The Qwen tables behave alike.Source: §C.2 · p. 14
  • Value projections are nearly isotropic, with effective rank 230–252 of 256 on DeepSeek and 154 of 160 on Qwen; projection weighting raises row error by only about 1%.The head gain τj also varies by n-gram order: the same error matters 2.3 times more in a block 2 2-gram head and 2.9 times more in a block 15 2-gram head than in a 4-gram head.Source: §C.2 · p. 14, Table 6 · p. 14
  • The gate-weighted key projection is almost rank one on DeepSeek (effective rank 1.2–2.4) and has effective rank 52.5 on Qwen; its ten leading directions hold 98–100% of DeepSeek’s energy.Quantized rows’ key-weighted error is 0.17–0.26 times their unweighted error, helping explain why gate error is small.Source: §C.2 · p. 14, Table 6 · p. 14

C.3 Gates Computed from Level 1

The analyses find that level-1 gates closely track FP8 gates, while lookup importance tends to oppose access frequency and concentrates in context-dependent subsets. These patterns distinguish gate-based allocation from row-frequency proxies.

C.3 Gates Computed from Level 1

  • On PopQA, quantizing only gates changes accuracy by +0.08 to +0.15 points, while the values-only and both-quantized losses differ by −0.16, +0.08, and −0.04 points across three depths.Every interval for the values-only versus both-quantized difference contains zero; the table reports changes from FP8 52.39 over 14,267 questions.Source: §C.3 · p. 15, Table 9 · p. 15

C.4 Access Frequency, Gate, and Importance

  • Rows accessed more than 1,000 times account for 32% of held-out English 2-gram lookups but only 4.5% and 1.9% of importance in blocks 2 and 15; rows accessed at most 100 times carry 79% and 88%.On PopQA, never-seen rows serve 14% of lookups and carry 23% of importance, while rows seen more than 1,000 times serve 32% and carry 3.8%.Source: §C.4 · p. 15, Table 10 · p. 15
  • Gate and row access count have negative Spearman correlation in all 18 layer, order, and corpus combinations, ranging from −0.63 to −0.02.The relationship is strongest for 2-gram heads and code, and weakest for 4-gram heads and Chinese.Source: §C.4 · p. 15, Table 11 · p. 16
  • Importance is concentrated in a small set of lookups, but a row’s gate varies across its accesses: rows seen at least 20 times cover only 9% of held-out English lookups and have median p90/p10 gate-energy ratios of 2.0–2.1 in block 2 and 4.3–8.8 in block 15.On PopQA, the top 1% of lookups in block 15 carry 54% of importance.Source: §C.4 · p. 16, Table 12 · p. 16

D Residual Ladder

The residual ladder refines each row one level at a time, with each additional level reducing but not eliminating error. Its distortion is close to that of direct codes at similar rates, with the reported gap attributed to the quantizers rather than nesting.

D Residual Ladder

  • TCQ leaves 26.9% of DeepSeek’s energy after level 1, 7.2% after level 2, and 1.9% after level 3, with each level removing less error than the preceding one.The errors and prices for the two models are close.Source: D Residual Ladder · p. 16, Table 13 · p. 17
  • Because each level removes less error, the price θk of later levels rises, and the lookup sets selected to read each level are nested.For DeepSeek’s TCQ ladder, refining from level 1 to 2 is worthwhile above importance 5.4µ, and from level 2 to 3 above 20.1µ.Source: D Residual Ladder · p. 16, Table 14 · p. 17
  • On DeepSeek, two TCQ levels leave 7.2% error at 2.13 bits versus 6.86% at 2.06 bits for a direct two-bit code; three levels leave 1.9% at 3.19 bits versus 1.77% at 3.06 bits for a direct three-bit code.Two-level PQ leaves 10.6% error and two-level INT leaves 12.3% at 2.50 bits, versus 9.72% and 10.76% for direct codes at 2.25 bits, respectively.Source: D Residual Ladder · p. 17

E Read Budgets and Bit Accounting

The read budgets count logical bits per looked-up element, including scales, and specify how many lookups read deeper levels. The accuracy comparisons use the same read fractions across selection rules, while stored table size can depend on the rule.

E Read Budgets and Bit Accounting

  • A TCQ or PQ level costs 1.0625 bits per element on DeepSeek and 1.10 on Qwen, including a 16-bit row scale; an INT level costs 1.25 bits.The reported logical rate includes the scale of every level read, though physical bytes transferred can be higher.Source: E Read Budgets and Bit Accounting · p. 17
  • The 1.50-bit plan reads level 2 for 41.18% of DeepSeek lookups and 36.36% of Qwen lookups; the 2.10-bit plan adds level 3 for 37.65% and 34.9%, respectively.At 2.10 bits, one split between deeper levels is used across rules, codes, and tasks.Source: E Read Budgets and Bit Accounting · p. 17, Table 15 · p. 17
  • GateRQ and the Random rule store the full ladder, while Prior and Frequency need deeper levels only for rows their row-based rules can select.The accuracy tables report bits read per lookup, not the total bits stored.Source: E Read Budgets and Bit Accounting · p. 17

F.1 Native Implementation

The section describes GateRQ’s vLLM implementation and evaluation, then reports that gate-ranked reads outperform alternative selection rules across ladders and tasks.

F.1 Native Implementation

  • Level 1 stays on GPU while deeper records are shared from pinned host memory; the native serving path is synchronous and intended for prefill blocks.Below about 300–500 tokens, the fixed exchange latency makes it slower; physical transfers can exceed the logical read rate.Source: §F.1 · p. 18

F.2 Evaluation Protocol

  • Evaluation uses four benchmarks with matched prompts and greedy decoding, comparing Gate, Prior, Frequency, and Random using paired question-bootstrap intervals.Each configuration ran once, so intervals capture question sampling, not between-run variation.Source: §F.2 · pp. 18–19, Table 17 · p. 19

G.1 INT and PQ Ladders

  • Across all 12 model–code–budget settings, GateRQ has the best PopQA accuracy and four-task average among the four rules.The settings span two models, three codes, and two read budgets.Source: §G.1 · p. 19

G.2 Gate Against the Selection Rules

  • GateRQ beats Prior, Random, and Frequency in all 36 PopQA comparisons by 0.39–2.86 points, with every interval excluding zero.In nine of 12 settings, reading the next uniform-ladder level is not significantly different; in three, it is 0.34–0.36 points better at 0.70–1.36 more bits per lookup.Source: §G.2 · p. 19
  • On TriviaQA and CMMLU, GateRQ is significantly better in 20 and 21 of 36 comparisons; rules are harder to separate on NQ-open.The sole significant exception is DeepSeek TriviaQA with TCQ at 2.10 bits, where Frequency is 0.35 points better.Source: §G.2 · p. 19

G.3 Registered Predictions

The registered offline proxy is compared with measured PopQA changes and captures both prediction magnitudes and rule rankings across settings.

G.3 Registered Predictions

  • 57 of 66 registered PopQA-change predictions are within 0.4 points of measurement, including 21 of 22 TCQ predictions and 18 of 22 for each other code.The largest deviations for INT and PQ are 0.66 and 0.67 points, respectively.Source: §G.3 · pp. 19–20
  • The proxy predicts the four-rule order correctly in all 12 settings spanning two models, three codes, and two read budgets.It errs conservatively on DeepSeek by an average 0.22 points and by 0.09 points the other way on Qwen.Source: §G.3 · p. 20, Table 18 · p. 20
  • Five of the nine prediction deviations above 0.4 points are for Frequency.Predictions were written to files before each run queue.Source: §G.3 · pp. 19–20

H Serving Throughput

Native DeepSeek serving reduces GPU table memory and improves long-context throughput relative to GPU-resident FP8, with workload-specific measurements and accuracy still to cross-check.

H Serving Throughput

  • 11%–31% higher long-context throughput accompanies GPU table memory falling from 23.6 to 3.04 GiB per GPU versus FP8.vLLM reports about 8.1× greater KV-cache capacity; the throughput gain over GPU-resident FP8 comes from the larger KV cache.Source: H Serving Throughput · p. 20, Table 22 · p. 22
  • Against FP8 host offloading, throughput is similar at 1.50–2.10 bits per lookup while pinned host memory falls from 188.7 to 25.8–48.6 GiB.The comparisons use 32K-token inputs and 1K- or 2K-token outputs at 64 or 128 concurrent requests.Source: H Serving Throughput · pp. 20–21, Table 22 · p. 22
  • The serving measurements are workload-specific single runs, and native four-task accuracy remains to be cross-checked.The 1.50-bit setting varied by at most 0.4% and 0.8% between the three nodes at the two workloads.Source: H Serving Throughput · pp. 20–21

Related papers

Loading 2610.35985v1…