PaperlayerView exact revision on arXiv

Source-linked AI summary

AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, Kehong Yuan

arXiv:2501.15021v1cs.CL

TL;DR

Long multimodal inputs make VLM KV caches large, and LLM-oriented quantization overlooks multimodal attention-saliency differences. AKVQ-VL uses TSA- and PSA-aware adaptive quantization with WHT to reduce KV outliers. Across 12 long-context and multimodal tasks, it maintains or improves performance while reducing peak memory usage by 2.13×.

  • Problem

    Long multimodal inputs create oversized KV caches, while LLM-oriented quantization methods overlook VLM attention-saliency differences.

    Source: §I · p. 1

  • Method

    AKVQ-VL uses TSA- and PSA-based salient-token identification with adaptive mixed-precision KV quantization and WHT to reduce outliers.

    Source: §I · p. 2, B. Attention-Aware Salient Token… · p. 4

  • Results

    Across 12 long-context and multimodal tasks, AKVQ-VL maintains or improves performance, and peak memory usage falls by 2.13×.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results show that AKVQ-VL can compress VLM KV caches to 2 bits while preserving or improving downstream performance across the evaluated tasks.

    Source: B. Main Results · p. 6, §V · p. 7

Abstract

from arXiv · show

Vision-language models (VLMs) show remarkable performance in multimodal tasks. However, excessively long multimodal inputs lead to oversized Key-Value (KV) caches, resulting in significant memory consumption and I/O bottlenecks. Previous KV quantization methods for Large Language Models (LLMs) may alleviate these issues but overlook the attention saliency differences of multimodal tokens, resulting in suboptimal performance. In this paper, we investigate the attention-aware token saliency patterns in VLM and propose AKVQ-VL. AKVQ-VL leverages the proposed Text-Salient Attention (TSA) and Pivot-Token-Salient Attention (PSA) patterns to adaptively allocate bit budgets. Moreover, achieving extremely low-bit quantization requires effectively addressing outliers in KV tensors. AKVQ-VL utilizes the Walsh-Hadamard transform (WHT) to construct outlier-free KV caches, thereby reducing quantization difficulty. Evaluations of 2-bit quantization on 12 long-context and multimodal tasks demonstrate that AKVQ-VL maintains or even improves accuracy, outperforming LLM-oriented methods. AKVQ-VL can reduce peak memory usage by 2.13x, support up to 3.25x larger batch sizes and 2.46x throughput.

I. INTRODUCTION

Long multimodal sequences make VLM KV caches costly, while LLM-oriented quantizers overlook VLM-specific attention saliency. AKVQ-VL uses TSA- and PSA-guided adaptive quantization with WHT to address this mismatch.

I. INTRODUCTION

  • Long, redundant multimodal sequences create oversized KV caches and memory and I/O bottlenecks, while LLM quantizers overlook VLM attention differences.The paper says this mismatch can lead to suboptimal downstream performance and notes that prior KV quantization research had not specifically addressed it.Source: §I · p. 1
  • Initial VLM layers prioritize text tokens (TSA), then shift to PSA, where a few pivot tokens dominate attention, including tokens within vision tokens.This contrasts with approaches focused on retaining only the initial few tokens.Source: Figure 1 · p. 1, §I · p. 2
  • AKVQ-VL identifies salient tokens from TSA and PSA, quantizes them at higher precision, and assigns 2 bits to remaining tokens; WHT reduces KV outliers.The paper presents AKVQ-VL as the first approach specifically designed for VLM KV-cache quantization.Source: §I · p. 2
  • The authors analyze attention across VLM and LLM layers and heads, identifying TSA and PSA and their transition as guidance for VLM KV-cache compression.They also describe WHT-based outlier-free KV caches as facilitating extremely low-bit quantization.Source: §I · p. 2
  • Across 12 long-context and multimodal tasks, AKVQ-VL maintains or improves performance; peak memory usage falls by 2.13×.It also supports up to 3.25× batch sizes and boosts throughput by 2.46× on LLaVA-v1.5-7B.Source: §I · p. 2

II. BACKGROUND

VLM inference has prefill and decoding stages, with attention layers computing and caching Key-Value pairs. During prefill, input embeddings are transformed using Key and Value weight matrices.

A. VLM Inference with multimodal KV Cache

1) Prefill Phase:

  • In prefill, the model processes the prompt token sequence and generates the initial output token while computing KV pairs in each attention layer.The input embeddings X have length l_prompt and model hidden size d.Source: 1) Prefill Phase · p. 2
  • The Key and Value calculations use weight matrices Wk and Wv, respectively.The supplied figure caption describes average attention scores in LLaVA, with TSA in the first two layers and PSA in other layers.Source: 1) Prefill Phase · p. 3, Figure 3 · p. 3

2) Decoding Phase:

During decoding, each attention layer processes one input token, computes its Key and Value, and updates the KV cache. The methodology analyzes VLM and LLM attention patterns and combines salient-token identification with WHT-based quantization.

2) Decoding Phase:

  • Each decoding step takes a single token as input and computes its Key and Value in each attention layer.The resulting tK and tV update the KV cache, which supports subsequent computations.Source: 2) Decoding Phase · p. 3
  • The multimodal KV cache stores historical information from both vision-encoder tokens and tokenizer text tokens.Unlike an LLM, a VLM handles tokens from both modalities.Source: 2) Decoding Phase · p. 3

III. METHODOLOGY

  • AKVQ-VL combines attention-aware salient-token identification with outlier-free adaptive KV-cache quantization using WHT.The attention analysis compares VLMs with LLMs; prompts contain text and multiple images from MileBench, with LLaVA-v1.5-7B as the VLM case study and LLaMA2-7B as the comparison.Source: §III · p. 3, A. Distinct Attention Patterns in VLM · p. 3

1) Local Attention Pattern:

VLM attention is local, initially dispersed, and later concentrated on a small set of tokens. The analysis identifies text-salient attention in early layers and pivot-token saliency in later layers.

1) Local Attention Pattern:

  • Attention in both VLMs and LLMs focuses more on recent tokens, while shifting from dispersed attention in early layers to concentration on a small token set later.Table I summarizes TSA and PSA patterns across several VLMs.Source: 1) Local Attention Pattern · p. 3, Table I · p. 3

2) TSA Pattern:

  • In the first two attention layers, text tokens receive more attention than vision tokens, and most VLM attention heads show varying degrees of TSA.The comparison reports no significant attention difference between token groups in the LLM.Source: 2) TSA Pattern · p. 3

3) PSA Pattern:

  • In subsequent layers, pivot tokens dominate attention, and in VLMs they can occur away from the sequence beginning.The figure caption describes modality-wise average attention scores and excludes the first five tokens to mitigate sink-token influence.Source: 3) PSA Pattern · p. 3, Figure 4 · p. 4
  • The analysis distinguishes text and vision token behavior and identifies pivot-token occurrence patterns across multiple VLMs.The reported models include LLaVA-V1.5-13B, LLaVA-v1.6-vicuna-7b, LLaVA-v1.6-mistral-7b, and Qwen2-VL-7B-Instruct.Source: 3) PSA Pattern · p. 3

B. Attention-Aware Salient Token Identification

AKVQ-VL identifies salient tokens from TSA and PSA, assigns them higher precision, and quantizes remaining tokens to 2 bits. WHT mitigates outliers to support extremely low-bit quantization.

B. Attention-Aware Salient Token Identification

1) Salient Tokens of TSA:

  • For TSA, text tokens are treated as salient, alongside recently generated tokens selected using attention locality.This avoids reliance on attention scores during inference, which is less suitable for kernel-based attention acceleration such as FlashAttention.Source: 1) Salient Tokens of TSA · p. 4

2) Salient Tokens of PSA:

  • For PSA, massive activations identify token indices used to locate additional pivot tokens, which are salient alongside recent tokens.The passage relates these activations to concentrated attention in subsequent layers.Source: 2) Salient Tokens of PSA · p. 4

C. Outlier-Free Adaptive KV Cache Quantization with WHT

  • The cited text reports that WHT mitigates the impact of outliers and facilitates extremely low-bit quantization, without specifying that those outliers cause substantial quantization errors.The figure caption identifies Key magnitudes before and after WHT in LLaVA [11], layer 10, head 0.Source: C. Outlier-Free Adaptive KV Cache… · p. 4, Figure 5 · p. 4

1) Adaptive Per-Token Dynamic KV Cache Quantization:

AKVQ-VL assigns precision by salient-token category and uses WHT-based equivalent transformations to reduce KV outliers for low-bit quantization.

1) Adaptive Per-Token Dynamic KV Cache Quantization:

  • Pivot and recent tokens retain 16-bit KV values, while text tokens use 4-bit precision and per-token dynamic asymmetric quantization with clipping.The text-token precision is used as output length expands.Source: 1) Adaptive Per-Token Dynamic KV Cache… · p. 4

2) WHT-Based Outlier Reduction:

A. Experiment Settings

Experiments cover VLMs using both MHA and GQA, with accuracy evaluated on 12 multimodal tasks from MileBench.

1) Models:

  • Evaluations include LLaVA-v1.5-7B/13B and LLaVA-v1.6-vicuna-7B with MHA, plus LLaVA-v1.6-mistral-7B with GQA.The models span Multi-Head Attention and Grouped Query Attention.Source: 1) Models · p. 5

2) Tasks and Metrics:

  • 12 MileBench tasks spanning multi-image and long-context settings are evaluated using accuracy.Tasks include object, navigation, scene, webpage, textbook, multimodal, slide, and document understanding.Source: 2) Tasks and Metrics · pp. 5–6

3) Baselines:

Across 12 multimodal tasks and multiple VLMs, AKVQ-VL has the highest accuracy in most cases; the section also describes baselines and clip-ratio choices.

3) Baselines:

  • Baselines include FP16, RTN INT4/INT2, SmoothQuant, KIVI, and SKVQ, with quantization group size 128 for all methods.KIVI residual length and SKVQ window size are also set to 128.Source: 3) Baselines · p. 6

B. Main Results

  • Across 12 multimodal tasks and multiple VLMs, AKVQ-VL achieves the highest accuracy in most cases, often outperforming RTN INT4 and sometimes FP16 despite mostly 2-bit caches.The comparison also reports that it surpasses existing LLM-based approaches.Source: B. Main Results · p. 6

1) Clip Ratios of Quantization:

  • For Document QA and Webpage QA, 2-bit quantization performs best at clip ratios 0.7–0.8; selected ratios are 0.8 for 2-bit and 1.0 for 4-bit.The 4-bit results are less sensitive to clip ratio.Source: 1) Clip Ratios of Quantization · p. 6

2) Numbers of Recent Tokens and Pivot Tokens:

The paper reports parameter choices for recent and pivot tokens, component-ablation findings, efficiency gains, and its conclusion of nearly lossless 2-bit quantization.

2) Numbers of Recent Tokens and Pivot Tokens:

3) Ablation Study of the Proposed Components:

D. Efficiency Analysis

  • On LLaVA-v1.5-7B with approximately 500-token inputs, AKVQ-VL reduces peak memory 2.13×, supports up to 3.25× larger batches, and boosts throughput 2.46×.The tests used four NVIDIA V100 16GB GPUs and throughput may improve further with kernel fusion.Source: D. Efficiency Analysis · p. 6, Figure 9 · p. 7

V. CONCLUSION

  • The conclusion characterizes AKVQ-VL as the first VLM-specific KV-cache quantization approach, achieving nearly lossless 2-bit quantization while reducing memory and enhancing throughput.The method adaptively compresses multimodal KV caches and uses WHT to reduce outlier impact.Source: §V · p. 7

Related papers

Loading 2501.15021v1…