PaperlayerView exact revision on arXiv

Source-linked AI summary

RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models

Zhiping Wu, Dongdong Ren, Yangchengyu Zhou, Zhengjie Zhang, Wenbin Li, Hongbing Pan, Yang Gao

arXiv:2609.25492v1cs.CVcs.AI

TL;DR

VLM quantization remains challenging because unimodal PTQ relies on Euclidean error assumptions that do not account for cross-modal sensitivity and geometry. RGSQ uses a modality-aware Fisher–Riemannian metric and geometry-guided error steering; it improves weight-only and weight–activation quantization across model families, scales, and bit-widths, with gains most pronounced at aggressive low-bit settings.

  • Problem

    Euclidean reconstruction and gradient heuristics may fail to characterize cross-modal curvature and second-order sensitivity consistently, especially under low-bit error accumulation.

    Source: §I · p. 2, B. Post-Training Quantization for VLMs · p. 3

  • Method

    RGSQ builds a modality-aware unified metric from modality-partitioned statistics and sensitivity-based weights, then uses sparse Givens rotations to steer quantization errors.

    Source: §I · p. 2

  • Results

    RGSQ consistently improves weight-only and weight–activation quantization across model families, scales, and bit-widths, with gains most pronounced under aggressive low-bit settings.

    Source: B. Main Results · p. 8, §VI · p. 13

  • Takeaways & Limitations

    RGSQ remains effective as a plug-in enhancement for existing Euclidean PTQ solvers, while its gains are strongest at layers with substantial vision–text geometric divergence.

    Source: §VI · p. 13

    RGSQ adds moderate one-time calibration overhead from layer-wise sensitivity estimation and sparse rotation search.

    Source: §VI · p. 13

Abstract

from arXiv · show

Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ). However, most PTQ methods are designed for unimodal large language models (LLMs). These methods treat quantization errors as isotropic perturbations under the Euclidean assumption, which provides weak guidance on directions most sensitive to quantization in VLMs. Consequently, directly adapting unimodal PTQ approaches or solely employing modality-specific scaling often leads to uneven bit-width distribution and inconsistent performance in low-bit settings. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric. RGSQ identifies modality-specific sensitive directions via Riemannian manifold mappings built from modality-partitioned empirical Fisher factors and fused into a modality-aware Kronecker-structured metric. We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes. Finally, we apply a whitening transformation that maps the Riemannian objective to an equivalent Euclidean form, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions. Across an extensive and diverse set of mainstream VLM benchmarks, RGSQ achieves the highest accuracy and stability under extremely low-bit settings (W2A8 and W3A8). It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9% and surpasses single-modality improvements by up to 8.6%.

I. INTRODUCTION

VLMs’ growing size makes deployment costly, while PTQ methods built for unimodal LLMs can degrade when applied directly to vision–language models. RGSQ addresses cross-modal differences in sensitivity and error direction using a unified Riemannian metric and geometry-guided rotations.

I. INTRODUCTION

  • VLMs’ growing parameter counts and memory footprints make deployment on resource-constrained devices challenging.The introduction gives LLaMA-3 70B at approximately 140 GB under FP16 as an example.Source: §I · p. 1

A. Post-Training Quantization for LLMs

  • PTQ reduces memory bandwidth and computation without retraining, but common LLM methods calibrate using Euclidean reconstruction errors.Weight-only methods target memory and storage overhead; weight–activation methods also use low-precision tensor cores.Source: §I · p. 1, A. Post-Training Quantization for LLMs · p. 3

I. INTRODUCTION

  • Existing LLM PTQ methods can cause substantial accuracy degradation when transferred directly to VLMs, while existing VLM-aware approaches still overlook natural-gradient sensitivity information.RSAVQ addresses directional errors but does not model cross-modal geometry.Source: §I · pp. 1–2
  • VLMs process visual and language features with distinct statistics and unequal task-loss contributions, making uniform isotropic calibration prone to misallocating limited capacity.The paper motivates this as a source of reduced fidelity for sensitive features and degraded performance.Source: §I · p. 2
  • RGSQ combines modality-specific second-order statistics and sensitivity-based weights into a unified Kronecker–Fisher metric for VLM quantization.It also uses sparse Givens rotations to steer quantization noise toward low-sensitivity directions.Source: §I · p. 2

B. Post-Training Quantization for VLMs

VLM-specific PTQ methods account for modality differences, but most still rely on Euclidean reconstruction and gradient heuristics. The paper motivates geometry-aware quantization that models cross-modal curvature and steers errors while remaining compatible with standard PTQ computation.

B. Post-Training Quantization for VLMs

  • VLM-aware PTQ methods use modality-aware scaling or calibration, yet most still rely on Euclidean error models and gradient heuristics that may not capture cross-modal curvature and second-order sensitivity.The paper identifies unstable behavior at lower bit-widths and on more challenging multimodal tasks as a concern.Source: B. Post-Training Quantization for VLMs · p. 3

C. Geometry-Aware Quantization

  • The geometry-aware gap is to map modality-specific weights and activations into a comparable metric space, then control error directions and return to a standard Euclidean computation graph.The paper says error direction and accumulation affect final loss in the low-bit regime.Source: C. Geometry-Aware Quantization · pp. 3–4
  • RGSQ measures errors with a unified modality-aware Fisher–Riemannian metric and uses sparse Givens rotations to steer them toward geometry-insensitive axes.The paper describes Euclidean PTQ and scalar modality reweighting as special cases of its view.Source: C. Geometry-Aware Quantization · p. 4

A. PTQ for LLMs and VLMs

  • A typical VLM combines a visual encoder, projector, and LLM backbone, with visual and text tokens concatenated before autoregressive decoding.The tokens originate from visual features and text embeddings, respectively.Source: A. PTQ for LLMs and VLMs · p. 4
  • Common Euclidean PTQ minimizes linear-layer output reconstruction error on a calibration set, assuming Euclidean deviations track task loss and modalities share an isotropic scale.The paper states that both assumptions are violated in VLMs.Source: A. PTQ for LLMs and VLMs · p. 4

B. Information-Geometric View of Quantization Error

The paper uses information geometry to assess quantization perturbations through Fisher curvature rather than Euclidean distance alone. It motivates a modality-aware metric and directional noise steering for VLMs, with a Kronecker approximation to make curvature modeling tractable.

B. Information-Geometric View of Quantization Error

  • A Fisher Information Matrix provides a Riemannian metric for measuring how parameter perturbations affect output distributions and task loss.The paper motivates geometry-consistent perturbation costs rather than Euclidean distances alone.Source: B. Information-Geometric View of… · p. 4
  • For linear layers, a Kronecker-factored empirical Fisher approximation retains anisotropic sensitivity from inputs and gradients while reducing storage and manipulation cost.The stated complexity falls from O((doutdin)^2) to O(dout^2+din^2).Source: B. Information-Geometric View of… · p. 4
  • Visual and language tokens can have distinct activation ranges, outlier patterns, and gradient statistics, which a single Euclidean norm flattens.The Fisher–Riemannian metric exposes modality-dependent sensitive directions through curvature factors.Source: B. Information-Geometric View of… · p. 4

IV. METHOD

  • RGSQ aggregates modality-partitioned statistics with sensitivity-based weights and adds geometry-guided error steering to its unified metric.The comparison contrasts this with isotropic Euclidean PTQ and standard Riemannian methods that uniformly weight tokens.Source: Table I · p. 5, §IV · p. 5

A. Riemannian Modality-Aware Metric

RGSQ builds a layer-wise Kronecker–Fisher metric by combining modality-partitioned second-order statistics with sensitivity-based modality weights. This metric represents cross-modal differences in a shared geometric space.

A. Riemannian Modality-Aware Metric

  • RGSQ fuses modality-specific second-order factors, weighted by loss sensitivity, into a unified Kronecker–Fisher metric for each layer.The metric is constructed by online accumulation of modality-partitioned statistics in a single calibration pass.Source: A. Riemannian Modality-Aware Metric · p. 5
  • At a given layer, a modality with larger loss gradients is typically more sensitive to perturbations; its sensitivity score uses token-level gradient vectors and the ℓ1 norm to emphasize sparse, strong coordinates.Source: A. Riemannian Modality-Aware Metric · p. 5
  • Sensitivity-based weights depend on per-token sensitivity rather than raw token counts, making the metric invariant to the absolute visual-to-text token ratio.A token-count-balanced variant may be used when a modality has very few tokens but should not be neglected.Source: A. Riemannian Modality-Aware Metric · p. 5
  • The unified metric remains positive semidefinite because its factors are convex combinations of positive semidefinite matrices, yielding a well-defined Riemannian norm.The shared metric measures perturbation length across modalities while retaining their sensitivity weights.Source: A. Riemannian Modality-Aware Metric · p. 5

B. Riemannian Objectives and Whitening

RGSQ minimizes quantization perturbations under the unified Riemannian metric, then uses whitening to reduce that objective to a Euclidean form. The reduction permits standard PTQ solvers to be reused without adding runtime operators.

B. Riemannian Objectives and Whitening

  • Whitening preserves the metric’s anisotropic, loss-sensitive weighting while reducing the Riemannian PTQ objective to an equivalent Euclidean form.Standard Euclidean PTQ solvers can optimize the whitened weights, and whitening is used only during calibration.Source: B. Riemannian Objectives and Whitening · p. 6
  • The Riemannian weight objective minimizes the unified-metric norm of weight quantization error, expressible as a Frobenius reconstruction objective in whitened coordinates.A small diagonal damping term is added to covariance factors before matrix square roots to ensure invertibility and numerical stability.Source: B. Riemannian Objectives and Whitening · p. 6
  • Sensitivity-proportional allocation of calibration capacity empirically improves stability over a single unweighted Euclidean scale.Source: B. Riemannian Objectives and Whitening · p. 6
  • The optional global geometry regularizer preserves pairwise similarity structure of calibration-batch embeddings to stabilize calibration against local minima with large global distortions.The passage motivates this constraint for aggressive low-bit quantization, where small per-layer errors can compound and harm tasks dependent on cross-modal alignment.Source: B. Riemannian Objectives and Whitening · p. 6

C. Geometry-Guided Error Steering

RGSQ uses sparse rotations to reparameterize channel coordinates and shape low-bit perturbations under the unified geometry. It selects rotations by minimizing whitened quantization error, guided by sensitivity-weighted curvature importance.

C. Geometry-Guided Error Steering

  • Sparse rotations reparameterize channel coordinates to steer quantization noise away from high-curvature, loss-sensitive directions.The method applies an orthogonal input-channel transform composed of sparse Givens layers.Source: C. Geometry-Guided Error Steering · pp. 6–7
  • Channel importance combines modality-dependent loss sensitivity with channel-wise curvature, and the same scores guide clipping and scale calibration so more important directions incur smaller perturbations.Source: C. Geometry-Guided Error Steering · p. 7
  • RGSQ greedily searches discrete angles for disjoint channel pairs ordered by sensitivity-weighted importance, selecting rotations that minimize the whitened quantization error.The candidate angle grid is Θ = {0, π/16, . . . , π}.Source: C. Geometry-Guided Error Steering · p. 7
  • Sparse Givens rotations reduce curvature-update cost from O(d^3) for dense rotations to O(d) per rotation, with total complexity O(Kd^2) across K layers.Weight rotation is precomputed offline; activation rotation remains online but adds only 4K multiplications per channel.Source: C. Geometry-Guided Error Steering · p. 7

D. Overall Calibration Loss

RGSQ combines geometry-aware weight reconstruction, modality-weighted activation reconstruction, and optional global semantic regularization in its calibration objective. Its modality-specific covariance matrices encode directional structure beyond scalar reweighting.

D. Overall Calibration Loss

  • RGSQ’s calibration objective combines weight reconstruction under unified Riemannian geometry, modality-weighted activation reconstruction, and optional global semantic regularization.The regularizer stabilizes calibration-time hyperparameter selection.Source: D. Overall Calibration Loss · p. 7
  • Unlike scalar reweighting strategies such as MBQ, RGSQ uses a full covariance matrix for each modality to capture directional activation structure.Source: D. Overall Calibration Loss · p. 7
  • The unified Riemannian norm penalizes weight perturbations according to their impact on the loss landscape rather than their bare Euclidean magnitude.The metric encodes which weight directions are sensitive to which modality.Source: D. Overall Calibration Loss · p. 8

A. Experimental Setups

Experiments use paired image-caption calibration data, evaluate multiple VLM families across vision-language and video benchmarks, and compare against classical and VLM-specific quantization baselines.

Calibration dataset

  • Calibration uses ShareGPT4V’s refined COCO Caption collection, concatenating visual tokens and roughly 64–256 caption tokens in each model-formatted sample.Each sample contains one COCO image and a GPT-4V-generated detailed caption.Source: A. Experimental Setups · p. 8

Evaluation benchmarks

  • Evaluation covers OCR, visual perception, multimodal reasoning, and video generalization benchmarks through LMMs-Eval.The suite includes OCRBench, TextVQA, VizWiz, SEED-Bench, ScienceQA, MMMU, SEED-Video, and Video-MME.Source: A. Experimental Setups · p. 8

Models

  • The study tests LLaVA-OneVision, InternVL2, and Qwen2-VL, with compact and large variants spanning 7B to 72B.The evaluated variants are LLaVA-OneVision 7B/72B, InternVL2 8B/26B, and Qwen2-VL 7B/72B.Source: A. Experimental Setups · p. 8

Baselines

  • Comparisons include classical Euclidean PTQ and VLM-specific methods, using group-wise quantization with group size 128.For W4A8, activations are quantized symmetrically per token and weights symmetrically per output channel.Source: A. Experimental Setups · p. 8

Implementation details

  • Experiments run on NVIDIA A100-80GB and A800-80GB GPUs with Python 3.10, PyTorch 2.9.0, CUDA 12.8, and Transformers 4.45.0.Inference uses LMMs-Eval default settings; code is publicly available.Source: A. Experimental Setups · p. 8

B. Main Results

RGSQ improves low-bit quantization results across weight-only and weight–activation settings, with particularly strong comparisons against Euclidean baselines. The section also identifies the study’s ablation components and default hyperparameter table.

Weight-only quantization (W3A16)

  • Under W3A16, RGSQ improves LLaVA-OneVision’s average score over AWQ by +6.9% on 7B and +19.0% on 72B.On the 72B model, RGSQ reaches 74.1%, near FP16’s 74.3%, while AWQ drops −17.0% relative to RTN.Source: B. Main Results · p. 8, Table II · p. 9

Weight–activation quantization (W4A8)

  • Under W4A8, RGSQ raises InternVL2-26B to 74.5% and Qwen2-VL-72B to 78.0%, near the latter’s 78.1% FP16 score.Gains over SmoothQuant and RTN reach +17.0% in representative cases.Source: B. Main Results · p. 8
  • W4A8 results further support multimodal-aware quantization, particularly when activation quantization makes modality-sensitivity disparities more pronounced.On InternVL2-26B, SmoothQuant falls from 72.7% to 65.5%, below RTN.Source: B. Main Results · pp. 8–9

Ablation and hyperparameters

  • The ablation table names RMAM, ROW, and GGES as components studied on LLaVA-OneVision-7B; Table III gives default quantization hyperparameters.RMAM denotes modality-partitioned statistics with a unified Kronecker-Fisher metric; ROW denotes whitening-based Euclidean reduction; GGES denotes sparse-Givens error steering.Source: Table IV · p. 9, Table III · p. 9

C. Enhancing Baseline Quantizers via ROW

RGSQ strengthens existing PTQ solvers through whitening and improves low-bit VLM quantization, with ablations showing gains from its metric, whitening, and error steering components.

C. Enhancing Baseline Quantizers via ROW

D. Low-Bit Quantization for VLMs

E. Ablation Study

  • On LLaVA-OneVision-7B W4A8, RMAM raises 56.1% to 64.1%, ROW to 66.6%, and GGES to 68.4%, above FP16's 67.5%.The ablation adds the components in sequence, starting from modality-agnostic Euclidean PTQ.Source: E. Ablation Study · pp. 10–11

D. Low-Bit Quantization for VLMs

F. Generalization to Video Understanding

RGSQ shows preliminary gains on video benchmarks without video-specific calibration, while layer-wise diagnostics link its error reductions to modality divergence. Its calibration adds offline cost, and the conclusion identifies efficiency improvements as future work.

F. Generalization to Video Understanding

  • Without additional video calibration, RGSQ improves over MBQ on three benchmarks for InternVL2-8B and LLaVA-OneVision-7B, with video-specific gains of 1.6% to 3.1%.The authors frame these results as an initial indication of generalization, not video-specialized performance.Source: F. Generalization to Video Understanding · pp. 11–12

G. Layer-Wise Geometric Diagnostics

  • At L14, modality eigenspaces have cosine similarity 0.15 and eigenvalue ratio 12.5×; the Riemannian metric reduces normalized quantization error by up to 78% at high-divergence layers.The reduction is smaller in the generation-heavy tail, where modalities are more aligned.Source: G. Layer-Wise Geometric Diagnostics · p. 12, Figure 5 · p. 12

H. Calibration Efficiency Analysis

VI. CONCLUSION AND LIMITATION

  • RGSQ improves weight-only and weight–activation quantization across model families, scales, and bit-widths, with a moderate one-time calibration overhead.The authors identify faster Fisher estimation, lighter rotation search, and finer token-level sensitivity modeling as future directions.Source: §VI · pp. 12–13

Related papers

Loading 2609.25492v1…