QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

1 INTRODUCTION

• MLA’s cache memory still scales linearly with context length and serving batch size, while existing approaches primarily quantize content to FP8 and retain high-precision RoPE keys.The functional impact of RoPE-path quantization is described as poorly understood, leaving joint low-bit compression largely unexplored.Source: §1 · pp. 1–2
• Across six models from four families, RoPE-path quantization causes greater attention-output distortion than content-path quantization under matched cache reconstruction error.The analysis attributes content-path effects to matching and aggregation, while RoPE-path effects enter through positional attention logits.Source: §1 · p. 2
• QuantMLA uses offline-fusible path-specific transformation spaces, with attention-output reconstruction for content and positional QK reconstruction for RoPE.The RoPE objective admits a theoretical bound on output distortion.Source: §1 · p. 2
• Across four MLA families, QuantMLA reports the first joint INT4 content-and-RoPE caching with minimal accuracy degradation, while INT2 content with INT4 RoPE remains competitive on reasoning and code benchmarks.The first-reported status is qualified as “to our knowledge.”Source: §1 · p. 3
• At 128K context, the native low-bit MLA backend provides up to 3.59× KV-cache compression and 5.168× BF16’s whole-job output throughput in the evaluated cache-pressure workload.The throughput comparison uses the same native vLLM scheduling policy.Source: §1 · p. 3

2.1 MLA DUAL-PATH CACHE FORMULATION

• MLA caches a shared content latent used for matching and aggregation, alongside a decoupled RoPE key used for positional matching.Only the content latent and RoPE key are cached.Source: §2.1 · p. 3

2.2 MATCHED CACHE ERROR, UNEQUAL OUTPUT DISTORTION

• Across six models, the RoPE-to-content ratio of model-level mean responses ranges from 1.941× to 6.864× under matched cache reconstruction error.Figure 2 shows layer-wise responses and RoPE/content ratios; layer-wise ratios vary substantially.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3 FUNCTIONAL ERROR ROUTES AND AMPLIFICATION

• Content-cache errors affect attention matching, value aggregation, and their interaction, while RoPE-cache errors affect only positional-logit matching.The section derives exact finite error routes for each path.Source: §2.3 · p. 4
• Equal-energy errors can differ in functional impact because amplification depends on average operator sensitivity, error-energy allocation, and signed cross-coordinate coupling.The model is validated through cross-layer prediction and controlled equal-energy interventions.Source: §2.3 · p. 4

3 QUANTMLA: FUNCTION-ALIGNED DUAL-PATH QUANTIZATION

• QuantMLA maps MLA’s dual-path asymmetry to path-specific offline-fusible transformation spaces, function-aligned objectives, and a native low-bit MLA backend.The transformations are fused into model parameters without online transformations.Source: §3 · p. 5

3.1 PATH-SPECIFIC OFFLINE-FUSIBLE TRANSFORMATION SPACES

• The content path uses an orthogonal rotation and positive channel scale that preserve full-precision computation and can be fused into model parameters offline.The shared transformed latent serves both matching and aggregation.Source: §3.1 · p. 5
• RoPE-compatible transformations use rotations and positive scales over frequency pairs, preserving positional scores while respecting RoPE’s pairwise rotational structure.The reciprocal factors fuse into pre-RoPE query and key projections without online transformations.Source: §3.1 · p. 5

3.2 FUNCTION-ALIGNED LEARNING OBJECTIVES

• Content-path attention-output reconstruction captures coupled matching and aggregation effects that score-only reconstruction would miss.It minimizes normalized post-output-projection distortion against a frozen BF16 teacher.Source: §3.2 · p. 6
• RoPE-path positional QK reconstruction targets positional-score errors, which admit a finite bound on output distortion.The content path and value representations remain fixed under this perturbation.Source: §3.2 · p. 6

Appendix C.3 provides the proof. The bound motivates normalized positional QK reconstruction:

• Using actual queries in positional QK reconstruction preserves cross-frequency cancellation, and minimizing its error controls the score-dependent term in the output-distortion bound.Equivalent query–key compensation preserves full-precision scores while leaving value representations unchanged.Source: Appendix C.3 · p. 6

3.3 NATIVE LOW-BIT MLA EXECUTION

• The decode kernel fuses unpacking and dequantization into attention without expanding per-head keys and values or materializing a full-history BF16 cache.Source: §3.3 · p. 6
• For INT4 history, attention uses BF16 for the first four sink tokens and most recent 128 tokens, and dequantized INT4 elsewhere, with all valid tokens in one global softmax.Non-sink tokens leaving the recent window switch to existing packed copies without additional quantization.Source: §3.3 · p. 6
• Separate metadata buffers enable earlier RoPE QK scheduling, while protected BF16 loads overlap with dequantization and reconstructed content tiles are reused across QK and aggregation.Synchronization precedes consumption, and buffers are overwritten only after prior consumers complete.Source: §3.3 · p. 6
• Table 1 reports percentage scores; CS averages five commonsense benchmarks, while Avg equally weights all ten tasks.Bold indicates the best quantized score within each precision, including ties.Source: Table 1 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation spans four MLA families, capability-matched task suites, and long-context retrieval, using C4R4 as the primary setting and C2R4 for further content-cache compression.Quantized methods use a shared per-token asymmetric affine quantizer with group size 64; learned transformations are calibrated offline.Source: §4.1 · p. 7

4.2 MAIN ACCURACY RESULTS

• At C4R4, QuantMLA stays within 0.26 and 0.13 percentage points of BF16 on DeepSeek-V2-Lite and Moonlight-16B-A3B, respectively.On GSM8K, DeepSeek-V2-Lite scores 35.25%, while Moonlight-16B-A3B matches BF16 at 74.53%.Source: §4.2 · p. 8
• At C4R4, QuantMLA exceeds the strongest same-precision baselines by 4.82 and 2.70 points on the reasoning-and-code suite for LongCat-Flash-Lite and GLM-4.7-Flash, respectively.At C2R4, GLM-4.7-Flash averages 51.43%, 6.22 points above the strongest same-precision baseline; task-to-task variation increases.Source: §4.2 · p. 8
• On S-NIAH-3 retrieval from 4K to 32K tokens, QuantMLA averages 87.20%, 7.40 points above the strongest quantized baseline, and leads quantized baselines at every tested length.It averages 99.95% and 99.60% on S-NIAH-1 and S-NIAH-2, respectively.Source: Figure 4 · p. 8, §4.2 · p. 9

4.3 ABLATION STUDIES

• On DeepSeek-V2-Lite MMLU, the C4R4 transformation sequence raises accuracy from 51.90% to 56.31% before BF16 protection; the full configuration reaches 57.95%, versus 57.90% for BF16.The content- and RoPE-path transformations add 1.22 and 3.19 percentage points, respectively; mixed precision adds a further 1.64 points.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 SYSTEM EFFICIENCY

• At 128K, C4R4 compresses persistent KV-cache storage by 3.59×; in the evaluated cache-pressure workload, whole-job throughput is 5.168× BF16.For five simultaneous requests with 128K input and 1K output tokens, throughput rises from 2.61 to 13.51 tokens/s, with zero C4R4 preemptions versus 32 for BF16.Source: §4.4 · p. 9, Figure 5 · p. 9

ETHICS STATEMENT

• The paper cautions that low-bit caching may alter model behavior despite preserved aggregate benchmark accuracy, and advises reassessing safety, calibration, and failure modes for the exact deployment configuration.The study uses public models and standard public benchmarks and involves no human subjects or personal-data collection.Source: ETHICS STATEMENT · p. 10

A.1 TRANSFORMATION-BASED LLM QUANTIZATION; A.2 LOW-BIT KV CACHE QUANTIZATION

• Prior quantization methods primarily target conventional explicit KV representations, while MLA’s shared content latent and decoupled RoPE keys impose distinct roles and transformation constraints.The related-work discussion identifies this architecture-specific asymmetry as the motivation for QuantMLA’s dual-path approach.Source: §A.1 · p. 15, §A.2 · p. 15

• The appendix supplements Section 2 with a matched-error protocol and six-model results, error-propagation and amplification derivations, and predictive and interventional validation.Source: B ADDITIONAL DUAL-PATH ERROR ANALYSIS · p. 15

B.1 MATCHED-ERROR PROTOCOL AND SIX-MODEL RESULTS

• The matched-error study perturbs one cache path at a time, holds the other path and model parameters fixed, and measures distortion after the attention output projection.INT4-induced error directions are rescaled to cache NMSE 0.01; the study covers six models, WikiText and C4 inputs, and 285 attention layers.Source: §B.1 · pp. 15–16
• Path responses are averaged across six dataset–seed cells before forming layer-wise RoPE-to-content ratios; model-level ratios use aggregate responses rather than averaging layer ratios.Figure 6 shows the 285-layer view, with output NMSE equal to 0.01 times the path response under the fixed cache NMSE.Source: §B.1 · p. 16, Table 4 · p. 16, Figure 6 · p. 16

B.2 ERROR PROPAGATION AND AMPLIFICATION

• Content-path perturbations affect both attention matching and the shared latent’s aggregation, with an interaction term that need not vanish; RoPE perturbations leave the content path unchanged.The finite-perturbation identities separate these routes without a linear approximation, and path-isolated learning does not assume independent routes.Source: §B.2 · p. 17
• Equal-energy errors can differ in functional impact because operator sensitivity, error-energy allocation, and signed cross-coordinate coupling all contribute to local response.The quantities are defined in the declared cache coordinates.Source: §B.2 · p. 17

B.3 PREDICTIVE VALIDATION AND CONTROLLED INTERVENTIONS

• The frozen predictor combines operator-sensitivity and quantization-error-allocation ratios with a fixed correction, and incorporating allocation improves prediction over trace-only and cache-error-only controls.The correction was set using an earlier DeepSeek-V2-Lite stage and not refitted on GLM-4.7-Flash.Source: §B.3 · p. 18, Table 5 · p. 18
• At DeepSeek-V2-Lite layer 20, permuting whole-token content errors increases gain by 5.918×, while sign randomization changes it little; broader permutations raise median content-path gain by 2.853× and 4.220× across two models.The interventions preserve error energy while changing its allocation.Source: §B.3 · pp. 18–19, Table 6 · p. 19

C.1 CONTENT-PATH FUSION

• The content transformation is absorbed into the producing and consuming parameters offline, preserving full-precision consumer outputs without an online transformation.The update applies to the content-latent sub-block; concatenated positional features remain unchanged.Source: §C.1 · p. 19

C.2 ROPE-PATH FUSION

• The RoPE transformation and reciprocal query compensation preserve full-precision positional scores and can be folded into pre-RoPE projections.The transformation follows native RoPE frequency-pair layouts and introduces no cross-frequency online mixing.Source: §C.2 · pp. 19–20

C.3 BOUNDING OUTPUT ERROR BY POSITIONAL QK ERROR

• The positional-QK output bound covers finite RoPE-score perturbations while holding queries, content scores, values, and valid-key sets fixed; content quantization adds terms.Equivalent query–key compensation preserves full-precision positional scores for each candidate transformation.Source: §C.3 · p. 20

C.4 QUANTIZATION AND TRANSFORMATION LEARNING

• QuantMLA learns content rotations, RoPE-pair rotations, and positive scales with pretrained parameters frozen, minimizing attention-output error for content and positional-QK error for RoPE.The RoPE objective uses the complete positional QK matrix over valid causal pairs and heads; checkpoint selection uses held-out path objectives.Source: §C.4 · p. 21

C.5 NATIVE CACHE LAYOUT AND MIXED-PRECISION EXECUTION

• The mixed-precision policy keeps sink and recent tokens in BF16 while retaining packed copies for all tokens; aging tokens switch to packed copies without requantization.Protected and quantized tokens contribute once each, with no historical token evicted or masked.Source: Figure 8 · p. 22, §C.5 · p. 23
• 3.59× persistent KV-cache compression at 128K (3.22× at 4K) includes BF16 protection buffers and redundant packed copies.The accounting excludes model weights, shared page tables, allocator overhead, and temporary workspaces.Source: §C.5 · p. 23

D.1 TASK EVALUATION AND SCORING

• Evaluation specifies task-specific prompting and scoring for reasoning, code, commonsense, and extraction benchmarks, with equal weighting of benchmarks in each suite average.Code generation includes single-sample pass@1, and bold main-table entries identify the best quantized result within each model, precision, and metric, including ties.Source: §D.1 · p. 24

D.2 BASELINE ADAPTATIONS

• SmoothQuant† uses calibrated statistical scaling, whereas QuaRot† uses fixed Hadamard rotations; both adapt to MLA’s native content and RoPE caches.QuaRot†’s post-RoPE transformation remains online, unlike the offline-fused transformations described for the content cache.Source: §D.2 · pp. 24–25

D.3 COMPONENT-ABLATION PROTOCOL

• The sequence rises from 51.90% for unprotected RTN to 53.12% with the content-path transformation, 56.31% with the RoPE-path transformation, and 57.95% with mixed-precision protection, versus 57.90% for BF16.The two transformation increments are measured without BF16-protected tokens; the final configuration protects four sink tokens and the most recent 128 tokens.Source: §D.3 · p. 25
• The content-path and RoPE-path transformation increments are measured before applying the mixed-precision cache policy.The final configuration retains four sink tokens and the most recent 128 tokens in BF16.Source: §D.3 · p. 25
• SmoothQuant† and QuaRot† are independent unprotected baselines, while the reported component gains follow the stated cumulative order.The gains should not be interpreted as order-independent effects.Source: §D.3 · p. 25

E ADDITIONAL ACCURACY RESULTS

• Table 8 breaks down the Commonsense averages into HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and WinoGrande for DeepSeek-V2-Lite and Moonlight-16B-A3B.The remaining task-level metrics are already reported individually in the main tables.Source: E ADDITIONAL ACCURACY RESULTS · p. 25, Table 8 · p. 26

F.1 NATIVE ATTENTION

• Native attention latency is measured for batches 8 and 32 at fixed contexts from 1K to 1M, with timing spanning attention-main through split-result combination.The setup uses synthetic BF16 queries and cache values; reported latency includes unpacking and dequantization but excludes cache writing and metadata operations.Source: §F.1 · pp. 25–26
• Native C4R4 output and log-sum-exp agree bit for bit with a BF16 reference using the same reconstructed mixed-precision cache and partitioning.These checks validate native execution, not equivalence to unquantized attention.Source: §F.1 · p. 26

F.2 SERVING UNDER CACHE PRESSURE

• The serving comparison runs DeepSeek-R1-0528 on eight GPUs with vLLM 0.10.2, using BF16 or C4R4 with sink4/local128 protection.Both configurations use FP8 weights, BF16 activations, disabled prefix caching, and at most 16 concurrent sequences.Source: §F.2 · p. 26
• Five simultaneous requests each use 131,072 input tokens and generate 1,024 output tokens; their combined 660,480-token budget exceeds BF16's 657,856 usable slots.BF16 repeatedly preempts and recomputes, while C4R4 accommodates the workload without preemption.Source: §F.2 · p. 26
• 5.168× higher whole-job output throughput than BF16 is reported for C4R4 under cache pressure, reflecting avoided recomputation rather than an attention-kernel speedup.Table 10 compares one C4R4 run with the recorded BF16 baseline; no confidence interval across independent runs is reported.Source: §F.2 · pp. 26–27, Table 10 · p. 27