Source-linked AI summary
MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models
Lulu Hu, Wenhu Xiao, Xin Chen, Xinhua Xu, Bowen Xu, Kun Li, Yongliang Tao
TL;DR
MLLMs challenge conventional PTQ because heterogeneous modalities produce mismatched activation scales and computational-invariance assumptions. MASQuant combines modality-specific smoothing with low-rank cross-modal compensation to retain a unified quantized weight structure. It performs consistently across evaluated vision-language and omni-modal benchmarks, with strong results under aggressive quantization settings.
Problem
Different modalities can have disparate activation magnitudes, causing unified channel-wise smoothing to over-smooth non-dominant modalities and produce severe quantization errors.
Method
MASQuant learns modality-specific smoothing factors and uses SVD-based low-rank compensation to preserve a unified quantized weight representation.
Results
MASQuant demonstrates consistent superiority over existing channel-wise smoothing PTQ methods across evaluated vision-language and omni-modal MLLM benchmarks.
Takeaways & Limitations
Modality-aware smoothing and cross-modal compensation support effective PTQ across both dual-modal and tri-modal MLLMs.
Abstract
from arXiv · showhide
Post-training quantization (PTQ) with computational invariance for Large Language Models~(LLMs) have demonstrated remarkable advances, however, their application to Multimodal Large Language Models~(MLLMs) presents substantial challenges. In this paper, we analyze SmoothQuant as a case study and identify two critical issues: Smoothing Misalignment and Cross-Modal Computational Invariance. To address these issues, we propose Modality-Aware Smoothing Quantization (MASQuant), a novel framework that introduces (1) Modality-Aware Smoothing (MAS), which learns separate, modality-specific smoothing factors to prevent Smoothing Misalignment, and (2) Cross-Modal Compensation (CMC), which addresses Cross-modal Computational Invariance by using SVD whitening to transform multi-modal activation differences into low-rank forms, enabling unified quantization across modalities. MASQuant demonstrates stable quantization performance across both dual-modal and tri-modal MLLMs. Experimental results show that MASQuant is competitive among the state-of-the-art PTQ algorithms. Source code: https://github.com/alibaba/EfficientAI.
1. Introduction
The introduction identifies smoothing misalignment as a fundamental obstacle to applying channel-wise PTQ in MLLMs and presents MASQuant to address it while preserving computational invariance.
- Visual, text, and audio tokens can have substantially different activation magnitudes, with visual activations typically 10–100× larger than text and audio.
- Unified channel-wise smoothing factors are dictated by the dominant modality, over-smoothing non-dominant modalities and causing severe quantization errors.
- MASQuant learns dedicated smoothing factors for each modality while using text-smoothed weights with modality-specific low-rank compensation during inference.
- Modality-Aware Smoothing optimizes modality-specific activation factors, while Cross-Modal Compensation uses the low-rank structure of inter-modal differences and SVD-based whitening.
- Across vision-language and omni-modal MLLMs, MASQuant demonstrates consistent superiority over existing channel-wise smoothing PTQ methods on evaluated benchmarks.
- The paper formalizes smoothing misalignment, proves inter-modal activation differences are low-rank, and presents MASQuant for vision-language and omni-modal MLLMs.
2. Related Work
Related work covers established LLM quantization strategies and emerging MLLM-specific methods, while identifying activation quantization as insufficiently addressed across modalities.
- LLM quantization includes QAT and PTQ, with PTQ methods using error compensation, channel-wise smoothing, rotations, or mixed precision.
- MLLM quantization faces cross-modal activation disparities, including visual token activations that can exceed textual activations by 20×.
- Existing MLLM methods address modality-specific quantization sensitivity, multimodal outliers, or weight-group scaling, but activation quantization remains inadequately addressed.
3. Preliminaries
The preliminaries describe computational-invariance PTQ, smoothing-based outlier reduction, and SVD-based whitening as foundations for unified multimodal quantization.
- PTQ maps floating-point tensors to low-precision N-bit integer tensors using scale, zero-point, rounding, and clipping operations.
- For a linear layer, computational invariance enables reformulation through a transformation matrix S that reduces activation outliers and improves quantization reconstruction loss.
- The notation WxAy distinguishes weight-only quantization, such as W4A16, from weight-activation quantization, such as W8A8 and W4A8.
- The paper applies modality-specific S transformations to seek robust and effective PTQ performance in MLLMs.
- 3.2. SVD-based Whitening: SVD-based whitening transforms activations and weights while truncated SVD minimizes reconstruction error for the transformed weights.
- 3.2. SVD-based Whitening: The paper positions whitening as a way to compensate for cross-modal weight differences and enable unified quantization across modalities.
4. MASQuant
MASQuant combines modality-specific smoothing with low-rank cross-modal compensation to address smoothing misalignment while retaining a single quantized weight representation.
- Framework: MASQuant frames its pipeline as Modality-Aware Smoothing followed by Cross-Modal Compensation for multimodal quantization.The framework is designed to address smoothing misalignment and cross-modal computational invariance.
- Modality-Aware Smoothing: Existing smoothing methods optimize a hyperparameter β, whereas MAS directly optimizes modality-specific diagonal smoothing matrices for greater flexibility.The optimized matrices contain smoothing factors for each modality and can discover patterns beyond β-parameterized formulas.
- Smoothing Misalignment: Unified smoothing factors are dominated by the largest modality-specific activation range, mismatching non-dominant modalities and degrading their quantization quality.The paper terms this failure mode smoothing misalignment and analyzes its SQNR degradation theoretically and empirically.
- Cross-Modal Compensation: MAS removes modality misalignment but creates modality-specific quantized weights, so CMC uses whitening and truncated SVD to obtain low-rank compensation.The correction bridges modality-specific weight differences while targeting the output reconstruction loss.
- Cross-Modal Compensation: CMC combines a text-smoothed base quantized output with modality-specific corrections, preserving one quantized weight for efficient inference.The complete MASQuant workflow is illustrated for a text-vision dual-modal setting with low-rank matrices L1 and L2.
- Evaluation: MASQuant is evaluated on multimodal and omni-modal benchmarks, including visual, audio, text, and joint text-audio-visual tasks.The reported tables compare MASQuant with existing quantization methods on multimodal and omni-modal MLLMs.
5. Experiments
Experiments show that MASQuant addresses modality dominance and smoothing misalignment across vision-language and omni-modal MLLMs, preserving accuracy while reducing compensation cost. Ablations support modality-specific smoothing, whitening-based low-rank compensation, and a two-epoch training choice.
- Quantization performance: MASQuant matches FP16 performance on Qwen2.5-VL models at W8A8, while RTN fails and SmoothQuant severely degrades at W4A8.The failure pattern indicates that dominant-modality smoothing becomes especially harmful under aggressive quantization.
- Quantization performance: At W4A8 on Qwen2.5-Omni-3B, SmoothQuant audio WER rises from 3.9 to 77.4 on Librispeech and from 7.5 to 94.2 on Wenetspeech, whereas MASQuant preserves near-FP16 quality.The results attribute audio collapse to its smaller activation magnitudes under vision- and text-dominated smoothing.
- Modality dominance: Visual tokens have larger activations than text in attention and MLP layers, causing smoothing factors to mismatch text distributions by orders of magnitude.The αi distributions are non-uniform across components, supporting modality-aware smoothing gains in both evaluated models.
- Cross-modal compensation: SVD-based whitening substantially reduces the effective rank of cross-modal weight differences across Qwen2.5-VL and Qwen2.5-Omni layers.This verifies the low-rank structure required by Cross-Modal Compensation.
- Ablation studies: Uniform smoothing produces 77.4 WER on LibriSpeech, while MAS achieves 3.8 WER under W4A8.The ablation exposes the weaker modality’s sensitivity to uniform smoothing across disparate activation ranges.
- Ablation studies: CMC surpasses MAS at rank ratio 0.08, approaches 3.5 SQNR, and requires one quarter of the rank needed by the non-whitened baseline to match MBQ.Text is selected as the base modality because alternatives would add computation and memory costs during decoding.
6. Conclusion
The conclusion identifies smoothing misalignment as the main obstacle to channel-wise smoothing in MLLMs and presents MASQuant as its solution. MAS and CMC provide modality-specific smoothing with a single quantized weight during inference.
- 6. Conclusion: MASQuant combines Modality-aware Smoothing and Cross-Modal Compensation to support modality-specific smoothing factors and a single quantized weight during inference.The method is demonstrated on multimodal benchmarks across the paper’s evaluated settings.