Source-linked AI summary
MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models
Lulu Hu, Wenhu Xiao, Xin Chen, Xinhua Xu, Bowen Xu, Kun Li, Yongliang Tao
TL;DR
MLLMs challenge conventional PTQ because heterogeneous modalities produce mismatched activation scales and computational-invariance assumptions. MASQuant combines modality-specific smoothing with low-rank cross-modal compensation to retain a unified quantized weight structure. It performs consistently across evaluated vision-language and omni-modal benchmarks, with strong results under aggressive quantization settings.
Problem
Different modalities can have disparate activation magnitudes, causing unified channel-wise smoothing to over-smooth non-dominant modalities and produce severe quantization errors.
Source: §1 · p. 1, §4.1 · p. 3Method
MASQuant learns modality-specific smoothing factors and uses SVD-based low-rank compensation to preserve a unified quantized weight representation.
Source: §1 · p. 1, §4.2 · p. 4Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Post-training quantization (PTQ) with computational invariance for Large Language Models~(LLMs) have demonstrated remarkable advances, however, their application to Multimodal Large Language Models~(MLLMs) presents substantial challenges. In this paper, we analyze SmoothQuant as a case study and identify two critical issues: Smoothing Misalignment and Cross-Modal Computational Invariance. To address these issues, we propose Modality-Aware Smoothing Quantization (MASQuant), a novel framework that introduces (1) Modality-Aware Smoothing (MAS), which learns separate, modality-specific smoothing factors to prevent Smoothing Misalignment, and (2) Cross-Modal Compensation (CMC), which addresses Cross-modal Computational Invariance by using SVD whitening to transform multi-modal activation differences into low-rank forms, enabling unified quantization across modalities. MASQuant demonstrates stable quantization performance across both dual-modal and tri-modal MLLMs. Experimental results show that MASQuant is competitive among the state-of-the-art PTQ algorithms. Source code: https://github.com/alibaba/EfficientAI.
1. Introduction
The introduction identifies smoothing misalignment as a fundamental obstacle to applying channel-wise PTQ in MLLMs and presents MASQuant to address it while preserving computational invariance.
- Visual, text, and audio tokens can have substantially different activation magnitudes, with visual activations typically 10–100× larger than text and audio.Source: §1 · p. 1
- Unified channel-wise smoothing factors are dictated by the dominant modality, over-smoothing non-dominant modalities and causing severe quantization errors.Source: §1 · p. 1
- MASQuant learns dedicated smoothing factors for each modality while using text-smoothed weights with modality-specific low-rank compensation during inference.Source: §1 · p. 1
- Modality-Aware Smoothing optimizes modality-specific activation factors, while Cross-Modal Compensation uses the low-rank structure of inter-modal differences and SVD-based whitening.Source: §1 · p. 2
- Across vision-language and omni-modal MLLMs, MASQuant demonstrates consistent superiority over existing channel-wise smoothing PTQ methods on evaluated benchmarks.Source: §1 · p. 2
- The paper formalizes smoothing misalignment, proves inter-modal activation differences are low-rank, and presents MASQuant for vision-language and omni-modal MLLMs.Source: §1 · p. 2
2. Related Work
Related work covers established LLM quantization strategies and emerging MLLM-specific methods, while identifying activation quantization as insufficiently addressed across modalities.
- LLM quantization includes QAT and PTQ, with PTQ methods using error compensation, channel-wise smoothing, rotations, or mixed precision.Source: §2 · p. 2
- MLLM quantization faces cross-modal activation disparities, including visual token activations that can exceed textual activations by 20×.Source: §2 · p. 2
- Existing MLLM methods address modality-specific quantization sensitivity, multimodal outliers, or weight-group scaling, but activation quantization remains inadequately addressed.Source: §2 · p. 2
3. Preliminaries
The preliminaries describe computational-invariance PTQ, smoothing-based outlier reduction, and SVD-based whitening as foundations for unified multimodal quantization.
- PTQ maps floating-point tensors to low-precision N-bit integer tensors using scale, zero-point, rounding, and clipping operations.Source: §3.1 · p. 2
- For a linear layer, computational invariance enables reformulation through a transformation matrix S that reduces activation outliers and improves quantization reconstruction loss.Source: §3.1 · p. 2
- The notation WxAy distinguishes weight-only quantization, such as W4A16, from weight-activation quantization, such as W8A8 and W4A8.Source: §3.1 · p. 2
- The paper applies modality-specific S transformations to seek robust and effective PTQ performance in MLLMs.Source: §3.1 · p. 2
3.2. SVD-based Whitening
- SVD-based whitening transforms activations and weights while truncated SVD minimizes reconstruction error for the transformed weights.Source: §3.2 · p. 2
- The paper positions whitening as a way to compensate for cross-modal weight differences and enable unified quantization across modalities.Source: §3.2 · p. 2
4. MASQuant
MASQuant combines modality-specific smoothing with low-rank cross-modal compensation to address smoothing misalignment while retaining a single quantized weight representation.
Framework
- MASQuant frames its pipeline as Modality-Aware Smoothing followed by Cross-Modal Compensation for multimodal quantization.The framework is designed to address smoothing misalignment and cross-modal computational invariance.Source: §4 · p. 3
Modality-Aware Smoothing
- Existing smoothing methods optimize a hyperparameter β, whereas MAS directly optimizes modality-specific diagonal smoothing matrices for greater flexibility.The optimized matrices contain smoothing factors for each modality and can discover patterns beyond β-parameterized formulas.Source: §4.1 · p. 3, §4.2 · p. 4
Smoothing Misalignment
- Unified smoothing factors are dominated by the largest modality-specific activation range, mismatching non-dominant modalities and degrading their quantization quality.The paper terms this failure mode smoothing misalignment and analyzes its SQNR degradation theoretically and empirically.Source: §4.1 · p. 3, §4.2 · p. 4
Cross-Modal Compensation
- MAS removes modality misalignment but creates modality-specific quantized weights, so CMC uses whitening and truncated SVD to obtain low-rank compensation.The correction bridges modality-specific weight differences while targeting the output reconstruction loss.Source: §4.3 · p. 5
- CMC combines a text-smoothed base quantized output with modality-specific corrections, preserving one quantized weight for efficient inference.The complete MASQuant workflow is illustrated for a text-vision dual-modal setting with low-rank matrices L1 and L2.Source: §4.3 · p. 5, Figure 3 · p. 5
Evaluation
- MASQuant is evaluated on multimodal and omni-modal benchmarks, including visual, audio, text, and joint text-audio-visual tasks.The reported tables compare MASQuant with existing quantization methods on multimodal and omni-modal MLLMs.Source: Table 1 · p. 6, Table 2 · p. 6
5. Experiments
Experiments show that MASQuant addresses modality dominance and smoothing misalignment across vision-language and omni-modal MLLMs, preserving accuracy while reducing compensation cost. Ablations support modality-specific smoothing, whitening-based low-rank compensation, and a two-epoch training choice.
Quantization performance
- MASQuant matches FP16 performance on Qwen2.5-VL models at W8A8, while RTN fails and SmoothQuant severely degrades at W4A8.The failure pattern indicates that dominant-modality smoothing becomes especially harmful under aggressive quantization.Source: §5.2 · p. 7
- At W4A8 on Qwen2.5-Omni-3B, SmoothQuant audio WER rises from 3.9 to 77.4 on Librispeech and from 7.5 to 94.2 on Wenetspeech, whereas MASQuant preserves near-FP16 quality.The results attribute audio collapse to its smaller activation magnitudes under vision- and text-dominated smoothing.Source: §5.3 · p. 7
Modality dominance
- Visual tokens have larger activations than text in attention and MLP layers, causing smoothing factors to mismatch text distributions by orders of magnitude.The αi distributions are non-uniform across components, supporting modality-aware smoothing gains in both evaluated models.Source: §5.4 · p. 7
Cross-modal compensation
- SVD-based whitening substantially reduces the effective rank of cross-modal weight differences across Qwen2.5-VL and Qwen2.5-Omni layers.This verifies the low-rank structure required by Cross-Modal Compensation.Source: §5.4 · p. 7, Figure 5 · p. 7
Ablation studies
- Uniform smoothing produces 77.4 WER on LibriSpeech, while MAS achieves 3.8 WER under W4A8.The ablation exposes the weaker modality’s sensitivity to uniform smoothing across disparate activation ranges.Source: §5.4 · p. 7
- CMC surpasses MAS at rank ratio 0.08, approaches 3.5 SQNR, and requires one quarter of the rank needed by the non-whitened baseline to match MBQ.Text is selected as the base modality because alternatives would add computation and memory costs during decoding.Source: §5.4 · p. 8
6. Conclusion
The conclusion identifies smoothing misalignment as the main obstacle to channel-wise smoothing in MLLMs and presents MASQuant as its solution. MAS and CMC provide modality-specific smoothing with a single quantized weight during inference.
6. Conclusion
- MASQuant combines Modality-aware Smoothing and Cross-Modal Compensation to support modality-specific smoothing factors and a single quantized weight during inference.The method is demonstrated on multimodal benchmarks across the paper’s evaluated settings.Source: §6 · p. 8