HyQuant: Hybrid-Precision Quantization for LLM Attention

Jiatong Ding; Bingxin Xing; Yu Zhang; Dian Ding; Xiaodong Yi; Xianbin Ouyang; Feihu Zhou; Kun Zhang; Zhenyu Guo; Hao Pan; Guangtao Xue; Yiming Zhang

Source-linked AI summary

HyQuant: Hybrid-Precision Quantization for LLM Attention

Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang

arXiv:2608.27875v1Aug 28, 2026cs.AI

TL;DR

Low-bit attention quantization can amplify errors because a few highly attended positions are more sensitive than others. HyQuant preserves those vertical-line positions and a local window in full precision while quantizing the remainder, achieving near-full-precision accuracy with reported decode speedups.

Problem
Uniform low-bit attention quantization ignores non-uniform sensitivity, over-compressing critical positions while wasting precision elsewhere.
Method
HyQuant retains vertical-line positions and a recent local window in full precision while quantizing the remaining attention states and KV cache to low-bit formats across Prefill and Decode.
Results
HyQuant maintains near-full-precision accuracy while achieving 1.32× to 3.58× decode-kernel speedup and 1.04× to 1.17× end-to-end decode speedup.
Takeaways & Limitations
The results support hybrid precision as an efficient design for preserving long-context understanding and mathematical reasoning while reducing attention costs.
Takeaways & Limitations
The experiments use NVIDIA H100 GPUs and do not evaluate larger models beyond Qwen3-32B because of GPU memory limitations.

Abstract

from arXiv · show

Quantization has been widely adopted in LLM training and inference to reduce cost and improve efficiency. However, low-bit quantization of the \emph{attention} module often introduces large errors at very low bit-widths, causing performance degradation. Existing methods mainly rely on smoothing techniques to handle outliers, while we propose a hybrid quantization design to better balance accuracy and efficiency. Specifically, we propose \textbf{HyQuant}, an efficient hybrid quantization framework for LLM attention. HyQuant quantizes most attention states into low-bit formats while retaining a small set of vertical-line tokens and local-window states in high precision. These accuracy-critical regions are selected using lightweight vertical-line-aware attention-pattern signals, reducing quantization error with limited overhead. In the Prefill stage, HyQuant uses a hybrid-precision quantized attention operator that preserves vertical-line tokens and a local sliding window in full precision while quantizing the remaining context. In the Decode stage, HyQuant applies the same principle to KV-cache compression and fuses KV dequantization with attention computation to improve memory and hardware efficiency. Across diverse tasks, models, and datasets, HyQuant maintains nearly lossless accuracy with an extremely simple design, demonstrating the efficiency and practical feasibility of hybrid quantization for LLM attention. Code is available at: https://github.com/jerrysfls/HyQuant .

1 Introduction

Long-context LLM inference faces compute-bound Prefill and memory-bound Decode bottlenecks, while uniform low-bit quantization over-compresses critical attention positions. HyQuant addresses this with hybrid precision across both stages and reports near-full-precision accuracy with decode speedups.

Motivation: Prefill is mainly compute-bound, whereas Decode is constrained by repeatedly reading and writing past KV states.These stage-specific bottlenecks motivate optimizing attention computation and KV-cache memory use separately.
Motivation: Uniform token-wise precision ignores non-uniform attention sensitivity, over-compressing high-contribution positions while wasting precision elsewhere.The paper frames this as a reasoning-centric mixed-precision design problem not addressed by existing token-homogeneous approaches.
Motivation: Vertical-line positions carry stable attention mass across long spans while typically covering less than 5% of tokens, motivating their full-precision retention.The pattern appears across layers, heads, and model families.
HyQuant: HyQuant retains persistent vertical-line positions and a recent sliding window in full precision while quantizing the remaining states to low bit-width.A lightweight detector identifies vertical lines, and the fixed window adds small overhead.
Results: 1.32× to 3.58× decode-kernel speedup and 1.04× to 1.17× end-to-end decode speedup accompany near-full-precision accuracy across four models and three benchmarks.The evaluation uses Qwen3-8B, Qwen3-32B, LLaMA3.1-8B, and GLM-4-9B with LongBench, GSM8K, and MATH500.

2 Motivation

Attention is highly concentrated on a small set of persistent vertical-line positions and recent local context, making uniform low-bit compression poorly matched to long-context inference. Error analyses show that retaining a tiny high-score subset in full precision substantially reduces quantization error.

Concentrated attention: Persistent vertical attention patterns appear across Qwen3-8B, Gemma4-31B, Qwen3.5-4B, and Llama3-8B.Bright vertical columns indicate positions repeatedly attended over long spans across layers and heads.
Attention-mass coverage: TOP-5%+WIN with W=128 covers over 80% of attention mass, showing that persistent high-score positions and recent local context dominate effective attention support.The comparison includes TOP-1%, TOP-5%, and their union with the local window.
Quantization error: Uniform quantization is mismatched to concentrated attention because errors at high-score positions are amplified through repeated KV access by many queries.The resulting quality degradation is consistent with the observed attention imbalance.
Quantization error: Retaining only the top-1% or top-5% high-score positions in full precision while quantizing the rest to 4-bit consistently approaches the 8-bit error level from 1K to 32K sequences.Uniform 4-bit quantization produces noticeably larger MSE than uniform 8-bit quantization.

3 Related Work

Prior work accelerates long-context inference through sparsification or quantization, but these approaches often incur overhead, quality loss, or token-homogeneous treatment. HyQuant instead assigns precision by attention importance and jointly optimizes Prefill and Decode.

Sparsification: Block-sparse and selective-retention methods can reduce Prefill computation, but preprocessing overhead may offset savings at short or moderate sequence lengths.These methods rely on online scoring, retrieval, or reordering to select computation or cache content.
Quantization: Quantized attention and KV-cache methods reduce bandwidth and storage, but aggressive bit-widths can introduce quality loss and require error-control strategies.Examples include smoothing or approximated attention in Prefill and asymmetric, sensitivity-aware, or outlier-aware KV quantization in Decode.
Difference from previous work: Unlike MInference, which uses vertical-line patterns as a sparsity mask, HyQuant uses them to assign different precisions while preserving less significant tokens in low-bit form.This retains information that a sparsity-only treatment overlooks.
Difference from previous work: HyQuant integrates Prefill and Decode optimization, using operand quantization in Prefill and operand plus cache quantization in Decode.The integrated design targets acceleration, memory reduction, and accuracy across both inference stages.

4 Method

HyQuant uses vertical-line-aware hybrid precision to retain a tiny set of attention-critical positions and a recent local window in full precision while quantizing the remaining majority. It applies fused hybrid operators in Prefill and Decode to reduce numerical error, KV memory traffic, and bandwidth pressure with limited overhead.

Core design: HyQuant retains persistent vertical-line positions and a recent sliding window in full precision, while quantizing the remaining attention states to low bit-width.The method targets the small set of positions carrying disproportionately large attention mass and dominating quantization error.
Vertical-line awareness: Vertical-line positions are selected from the non-window prefix using accumulated column-wise attention mass and a top-ρ rule.The score aggregates attention probabilities over a selected query set, and a lightweight reduction updates it with negligible overhead.
Prefill-stage hybrid attention: In Prefill, HyQuant splits keys and values into low-bit quantized and full-precision regions, then fuses both paths into one FlashAttention-like operator.The full-precision path covers KVL ∪ KWin, while the quantized path uses dequantized values and concatenation along the key dimension.
Decode-stage KV cache: In Decode, HyQuant stores the KV cache with full precision for KVL ∪ KWin(t) and low bits for the remaining majority, reducing KV memory footprint and bandwidth pressure.The method reserves precision for keys in the vertical-line set and recent window while compressing the non-window prefix.
Decode-stage fused attention: Decode dequantization is fused into the attention kernel, which dequantizes quantized KV blocks on the fly and updates online-softmax accumulators without materializing full-precision intermediates.Full-precision vertical-line and local-window blocks are loaded directly during the same scan.

5 Experiments

Experiments across four models and long-context reasoning benchmarks show that HyQuant preserves near-full-precision accuracy while improving efficiency over strict low-bit baselines. Its hybrid operators reduce Prefill error, accelerate Decode, and incur limited runtime and memory overhead.

Benchmark results: Across end-to-end benchmarks, HyQuant generally preserves FA2 performance and improves over applicable strict low-bit baselines in most settings.The comparison includes KIVI, KVTuner, and SageAttention; Qwen3-8B evaluation uses thinking mode.
Operator-level evaluation: HyQuant substantially reduces layer-wise Prefill attention-output MSE relative to SageAttention while retaining comparable Prefill latency.MSE is measured against full-precision FA2 at the intermediate attention output, using representative early layers.
Operator-level evaluation: 3.58× decode-kernel speedup is achieved over FA2 at a 32K prefix, while end-to-end Decode speedup is more moderate.The kernel comparison uses latency under different prefix lengths; speedup is computed relative to FA2.
High-parallel setting: Under batch size 32, HyQuant is the only compared method reported to operate, while fused dequantization improves speed in lower-batch settings.HyQuant unpacks KV values within attention without materializing the full-precision cache.
Benchmark results: HyQuant remains close to FA2 across evaluated benchmarks while avoiding the larger degradation observed with strict low-bit quantization.Slightly higher scores than FA2 on some datasets are treated as normal evaluation variance.
Ablation analysis: A larger local full-precision window slightly improves accuracy, supporting its complementarity with vertical-line-aware retention.The analysis attributes the improvement to retaining more recent tokens in full precision.
Overhead analysis: Vertical-line identification adds 3% to 5% of total runtime, while retaining 5% of vertical-line tokens increases non-window KV-cache size by about 15% versus strict 4-bit quantization.The total memory overhead also depends on local-window size.

6 Conclusion

HyQuant is a hybrid-precision framework for efficient long-context LLM inference. It retains selected vertical-line tokens and a local window in full precision, quantizes the remaining positions, and uses fused operators to achieve near-full-precision accuracy with faster Decode execution.

Conclusion: HyQuant identifies vertical-line tokens using accumulated column-wise attention scores, retains them and a local window in full precision, and quantizes the remaining majority.The framework applies this design to both Prefill attention and Decode KV-cache compression.
Conclusion: 1.32× to 3.58× decode-kernel speedup and 1.04× to 1.17× end-to-end decode speedup are achieved while maintaining near-full-precision accuracy.These results span multiple long-context and reasoning benchmarks.

Limitations

The evaluation is strongest for long-context settings, H100 GPUs, and models up to Qwen3-32B, while agent and coding-related effectiveness remains unknown.

HyQuant’s vertical-line retention has its clearest effect on long-context tasks; short-context gains may be smaller while remaining competitive with full precision.The authors identify short-context performance as a weaker setting for this mechanism.
On high-end GPUs, end-to-end speedups can be less pronounced at short context lengths because decoding is not fully memory-bound.
GPU memory limitations restrict evaluation to models no larger than Qwen3-32B, leaving larger models such as Qwen3-80B untested.
HyQuant’s effectiveness in agent and coding-related settings remains unknown despite assessment across tasks and models.

A Implementation Details and Algorithms

HyQuant combines lightweight vertical-line selection with fused hybrid-precision attention operators. It retains selected vertical-line tokens and a local window in full precision while quantizing the remaining KV states.

Vertical Token Identification: HyQuant identifies vertical-line tokens by excluding the local window, scoring prefix columns with tail-query proxies, and selecting the top-k positions.For GQA models, scoring uses a GQA-native form without explicitly materializing repeated KV heads.
Prefill Attention: The prefill operator partitions keys and values into a quantized prefix, full-precision vertical-line region, and full-precision local window.These regions are processed through a segmented FlashAttention-style online softmax.
Prefill Attention: The prefill quantized and full-precision paths are fused into one logical attention operator rather than separate attention calls.
Decode Attention: During decode, most historical KV states are stored in low-bit format and dequantized on the fly, while full-precision segments share the online softmax state.This avoids materializing a full-precision KV cache while retaining error-sensitive tokens.
KV-Cache Organization: At the prefill-to-decode boundary, HyQuant fixes the selected vertical-line set, preserves the local window, quantizes the remaining prefix, and stages newly generated tokens.The vertical-line set is computed once rather than recomputed at every decode step.
Evaluation Setup: The appendix evaluates HyQuant primarily on Qwen3-8B with K4V4 quantization and vertical-line-aware full-precision retention.

B.1 Math Reasoning Results

The appendix complements LongBench with math-reasoning evaluation and sensitivity studies. Larger local windows and higher vertical-line retention generally improve accuracy-related measures, with top-5% chosen as a practical trade-off.

Math Reasoning Results: HyQuant’s additional math-reasoning evaluation covers GSM8K and MATH500, complementing the main LongBench results.
Local Window Sensitivity: A larger local full-precision window slightly improves accuracy and reduces attention-output MSE.
Vertical-Line Ratio Sensitivity: Increasing the retained vertical-line token ratio generally improves average LongBench scores but increases the number of full-precision KV states.
Vertical-Line Ratio Sensitivity: The main experiments use top-5% vertical-line retention as a practical accuracy–efficiency trade-off.

B.4 Short-Context Evaluation

On short-context benchmarks, HyQuant remains competitive with full-precision attention and substantially outperforms strict 4-bit KIVI.

Short-Context Evaluation: HyQuant remains competitive with the full-precision baseline on short-context benchmarks.
Short-Context Evaluation: HyQuant substantially outperforms strict 4-bit KIVI on short-context benchmarks.

B.5 Component-Level MSE Ablation

HyQuant’s component ablation compares uniform quantization with local-window retention, vertical-line retention, and their combination using layer-wise MSE. The combined retention strategy achieves the lowest layer-wise MSE, while a separate ablation evaluates retaining long-tail tokens.

Component-Level MSE Ablation: Combining local-window and vertical-line retention produces the lowest layer-wise MSE.The study isolates each component before evaluating their combination.
Component-Level MSE Ablation: The ablation compares uniform quantization, local-window retention, vertical-line retention, and their combination on Qwen3-8B, with lower MSE preferred.Figure 6 reports component-level MSE under these four configurations.
Long-Tail Retention Ablation: A separate Table 12 ablation measures the marginal effect of retaining long-tail positions because vertical-only sparse selection may omit useful positions in long-context settings.HyQuant retains omitted long-tail positions in low precision, and the reported results demonstrate a benefit from keeping all tokens.