PaperlayerView exact revision on arXiv

Source-linked AI summary

MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression

Ofir Gordon, Ariel Lapid, Elad Cohen, Yarden Yagil, Arnon Netzer, Hai Victor Habi

arXiv:2507.09616v1cs.LGcs.CV

TL;DR

Transformer models are difficult to deploy on edge devices with limited memory and computational power. MLoRQ jointly optimizes low-rank approximation and quantization through per-layer rank and bit-width assignment under memory constraints. It reports state-of-the-art vision performance, including up to 15% accuracy improvement when compressing ViT-B below 12.5% of its original size.

  • Problem

    Transformer-based networks pose computational-complexity, power-consumption, and latency challenges for deployment on edge devices with limited memory and computational power.

    Source: §1 · p. 1

  • Method

    MLoRQ jointly optimizes low-rank approximation and quantization, using intra-layer candidate filtering followed by inter-layer rank and bit-width assignment.

    Source: §1 · p. 1

  • Results

    MLoRQ reports state-of-the-art vision performance and up to 15% accuracy improvement when compressing ViT-B below 12.5% of its original size.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results highlight the potential of combining low-rank approximation and quantization across tasks.

    Source: §7 · p. 10

    The compression-error formulation assumes the gradient is approximately zero and treats the layers’ contributions to compression-induced error as independent.

    Source: §4 · p. 5

Abstract

from arXiv · show

Deploying transformer-based neural networks on resource-constrained edge devices presents a significant challenge. This challenge is often addressed through various techniques, such as low-rank approximation and mixed-precision quantization. In this work, we introduce Mixed Low-Rank and Quantization (MLoRQ), a novel method that integrates both techniques. MLoRQ employs a two-stage optimization process to determine optimal bit-width and rank assignments for each layer, adhering to predefined memory constraints. This process includes: (i) an intra-layer optimization that identifies potentially optimal compression solutions out of all low-rank and quantization combinations; (ii) an inter-layer optimization that assigns bit-width precision and rank to each layer while ensuring the memory constraint is met. An optional final step applies a sequential optimization process using a modified adaptive rounding technique to mitigate compression-induced errors in joint low-rank approximation and quantization. The method is compatible and can be seamlessly integrated with most existing quantization algorithms. MLoRQ shows state-of-the-art results with up to 15\% performance improvement, evaluated on Vision Transformers for image classification, object detection, and instance segmentation tasks.

1. Introduction

Transformer models challenge deployment on edge devices with limited memory and computational power. MLoRQ jointly optimizes low-rank approximation and quantization, assigning each layer a rank and bit-width under memory constraints.

1. Introduction

  • MLoRQ targets minimal performance decrease under memory constraints, addressing the larger search space created by jointly considering rank and precision.The paper frames this challenge in the context of transformer deployment on edge devices with limited memory and computational power.Source: §1 · p. 1
  • Intra-layer search filters compression choices to a Pareto frontier, then inter-layer optimization assigns each layer a rank and bit-width.The framework also supports complementary quantization algorithms, with or without low-rank-aware adaptive rounding.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • An optional modified adaptive-rounding procedure refines weight quantization to mitigate compression-induced error from the combined low-rank and quantization representation.The method is described as compatible with most existing quantization schemes.Source: §1 · p. 2
  • MLoRQ reports state-of-the-art computer-vision performance and up to 15% accuracy improvement for ViT-B below 12.5% of its original size.The reported vision results cover ImageNet classification and COCO object detection.Source: §1 · p. 2
  • For BERT weight compression, MLoRQ reports an improvement of up to 7%.Source: §1 · p. 2

2. Background & Related Works

The background reviews low-rank approximation and quantization methods used in the work. It explains why matrix SVD suits transformer layers and introduces the uniform quantizer adopted here.

2.1. Low Rank Approximation

  • Transformer models mainly comprise fully connected layers and 1 × 1 convolutions, making matrix low-rank approximation via SVD a suitable choice.Source: §2.1 · p. 2
  • Weighted SVD methods add weighting to address cases where ordinary low-rank approximation does not yield optimal neural-network accuracy.The cited approaches use activation covariance or Fisher approximations to guide the decomposition.Source: §2.1 · p. 3

2.2. Quantization

  • The work uses a uniform quantizer, described as well supported by hardware, and identifies scale and zero point as its quantization parameters.The passage also lists MSE, min-max, and Hessian-MSE approaches for choosing those parameters.Source: §2.2 · p. 3

3. A Unified Low-Rank and Quantization Problem Definition

The paper formulates compression as choosing a rank and precision for each layer while minimizing performance degradation under a weight-memory constraint. Its motivation is that low-rank, quantization-only, and combined representations have different strengths across ranks and compression rates.

3. A Unified Low-Rank and Quantization Problem Definition

  • For fully connected layers, the compressed operation is hardware friendly and can be implemented as two consecutive fully connected operations.Source: §3 · p. 4
  • The problem is to choose each layer’s rank and precision using a representative dataset, minimizing performance degradation while satisfying a specified memory constraint.Inputs to a layer may already be perturbed by compression of preceding layers.Source: §3 · p. 4
  • No representation consistently gives the lowest MSE across ranks and compression rates, while the combined approach performs best at low ranks and high compression.The comparison considers quantization, low-rank approximation, and their combination.Source: §3 · p. 4

4. Mixed Low-Rank and Quantization

MLoRQ combines low-rank approximation and quantization through a two-stage search, with an optional rounding refinement. Its optimization separates per-layer candidate filtering from model-wide assignment under memory constraints.

4. Mixed Low-Rank and Quantization

  • MLoRQ filters per-layer compression candidates, assigns each layer a bit-width and rank, and optionally applies low-rank-aware adaptive rounding.The framework can complement nearly any quantization method.Source: §4 · pp. 4–5

4.1. Intra-Layer Search

  • Intra-layer search minimizes each layer’s compression error under a diagonal-Hessian assumption, sequencing low-rank decomposition, quantization-parameter selection, and candidate search.Source: §4.1 · p. 5

4.1.1. Hessian-aware SVD

  • Hessian-aware SVD addresses the layer approximation using an upper bound, with scaling integrated into output-channel quantization scales without adding noise.The method applies SVD to a Hessian-weighted weight matrix.Source: §4.1.1 · pp. 5–6

4.1.2. Quantization Parameters Selection

  • Quantization parameters are chosen for the original and decomposed weights; the A and B factor searches are separated to manage the large candidate set.The objective considers the SVD and Hessian factorization when assessing task-loss impact.Source: §4.1.2 · p. 6

4.1.3. Pareto Set Search

  • Pareto search retains options that are not dominated on both local error and memory footprint, forming the candidate set for each layer.Source: §4.1.3 · p. 6

4.2. Inter Layer Search

  • The inter-layer search selects one candidate per layer to minimize global error using SQNR, with ILP and interpolation for large rank-option sets.Source: §4.2 · p. 6

5. Low-Rank Aware Adaptive Rounding

LoRAda adapts adaptive rounding to weights jointly compressed by low-rank approximation and quantization. It sequentially optimizes each layer to reduce weight-perturbation and error-propagation effects.

5. Low-Rank Aware Adaptive Rounding

  • LoRAda modifies Adaptive Rounding for weights represented jointly through low-rank approximation and quantization.The procedure is designed for the combined compressed representation.Source: §5 · p. 7
  • For each layer, its sequence of optimizations aims to minimize errors from weight perturbations and their propagation through the model.Source: §5 · p. 7
  • The method defines continuous variables V_A and V_B for the sequence-mean mechanism, and uses soft-quantized low-rank factors e_A and e_B.Source: §5 · p. 7

6. Experiments

Experiments assess MLoRQ on image classification, COCO detection and segmentation, and GLUE, alongside component, compression-choice, and calibration-sample ablations. The reported comparisons include state-of-the-art image-classification results and gains on COCO and GLUE.

6. Experiments

  • Experiments cover ImageNet classification, COCO detection and instance segmentation, and GLUE language understanding, using Vision Transformers, Mask R-CNN variants, and BERT.Source: §6 · pp. 7–8
  • Unless specified otherwise, the setup uses weight bit-widths {2, 3, 4, 6, 8}, 1,024 optimization images, and 32 representative images for activation-parameter search and Hessian computation.Activations are uniformly quantized to 4-bit unless stated otherwise; compressed-weight memory is reported relative to full precision.Source: §6 · p. 8

6.1. Image Classification

  • MLoRQ achieves state-of-the-art results across image-classification architectures, with improvements of up to 15%.The section also reports that MLoRQ with mixed activation precision outperforms other PTQ methods.Source: §6.1 · p. 8

6.2. COCO Object Detection

  • For Mask R-CNN and Cascade Mask R-CNN, MLoRQ improves on state-of-the-art COCO results in multiple settings, including over 6 percentage points in AP bbox for Mask R-CNN.Source: §6.2 · p. 9

6.3. GLUE Benchmark Evaluation

  • On GLUE, MLoRQ achieves state-of-the-art results with a significant margin on nearly all tasks and compression rates.BERT-base models were fine-tuned per task before compression.Source: §6.3 · p. 9

6.4.1. Joint Low-Rank and Quantization Ablation

  • At high compression rates, adding low-rank options substantially improves accuracy over mixed-precision-only optimization for DeiT-B and ViT-L, while the effect is minimal for DeiT-T.Figure 4 shows better results than the quantization-only counterpart at the equivalence of 2-bit weight compression.Source: §6.4.1 · p. 9

6.4.2. MLoRQ Components Ablation

  • Each tested MLoRQ component independently improves performance, and their combination yields the best results.The ablation compares the full method with versions omitting or replacing individual components.Source: §6.4.2 · pp. 9–10, Table 5 · p. 10

6.4.3. Number of Samples Ablation

  • The 1,024-image calibration setting performs well, and increasing the sample count yields only marginal gains; MLoRQ also achieves a decent result with 32 images.Source: §6.4.3 · p. 10

7. Conclusions

The paper concludes that MLoRQ jointly assigns layerwise rank and bit-width under memory constraints, with adaptive-rounding refinement and reported improvements over state-of-the-art in vision. It situates the method among transformer-focused quantization and low-rank compression approaches, while identifying LLMs, VLMs, pruning, and one-bit quantization as future directions.

7. Conclusions

  • MLoRQ assigns rank and bit-width per layer through Pareto-based intra-layer candidate reduction and inter-layer allocation, followed by adaptive-rounding refinement.The conclusion describes integration into existing quantization schemes and reports notable improvement over state-of-the-art in vision tasks.Source: §7 · p. 10
  • The paper identifies extension to high-memory-demand LLMs and VLMs, and exploration of pruning and one-bit quantization, as future directions.Source: §7 · p. 10

A.1. Post-training Quantization for Transformers

  • Transformer PTQ work addresses activation outliers; RepQ-ViT [26] uses scale reparameterization and a Log2 quantizer, while ERQ targets activation and weight quantization error.Mixed-precision methods assign lower bit-widths to layers less sensitive to quantization.Source: §A.1 · p. 13

A.2. Low-rank Approximation for Model Compression

  • Low-rank compression factors weight matrices; FWSVD [18] and TFWSVD explore weighting strategies, while RankDyna dynamically assigns ranks by parameter importance.Source: §A.2 · p. 13

B. Local Error Analysis

The analysis compares low-rank-only, quantization-only, and joint representations across compression rates and ranks. Low-rank-only is ineffective below 25% compression but can help at larger compression ratios with minimal ranks.

B. Local Error Analysis

  • The study compares three representations—low rank only, quantization only, and their combination—using MSE across compression rates from 25% (8bit) to 6.25% (2bit).The test matrix Q = M1M2 has rank r, with factors sampled from a standard Gaussian distribution; quantization uses no clipping and low-rank decomposition uses SVD.Source: B. Local Error Analysis · p. 13
  • Low-rank representations are ineffective below 25% compression, but provide advantages at larger compression ratios with minimal ranks.Source: B. Local Error Analysis · p. 13
  • The analysis studies ranks r = 453 and r = 1000 and maps the optimal representation across compression rates and ranks.Source: B. Local Error Analysis · p. 13

C. Implementation Details

The implementation combines Hessian-aware weight decomposition and quantization-parameter search with integer optimization for weight and activation precision. It also describes correction and adaptive-rounding steps used to mitigate compression error.

C. Implementation Details

  • Experiments compute Label-Free Hessians, use Hessian-weighted SVD, and search percentile-based outlier thresholds for each weight output channel.The Hessian uses 100 iterations and batches of 32; the tested percentiles range from 0.97 to 1. Figure 6 analyzes layer error across compression rates.Source: C. Implementation Details · p. 13, Figure 6 · p. 14
  • After activation corrections, the method recomputes Hessian, SVD, and quantization parameters, then applies sequential adaptive rounding to reduce rounding error.The rounding optimization uses 20,000 Adam updates per layer; ViT-S uses a learning rate of 0.01, while the other stated rate is 0.3.Source: C. Implementation Details · p. 14

C.1. Mixed Precision Details

C.1.1. Activation Mixed Precision Details

  • Activation mixed precision uses maximal activation-tensor size as its memory constraint and a close-form search; Size(x) denotes the size of vector x.Source: §C.1.1 · p. 14

D. Search Space Analysis

The joint representation expands the candidate search space relative to low-rank and mixed-precision alternatives, with complexity scaling quadratically in bit-width options and linearly in ranks.

  • The joint representation has complexity quadratic in bit-width options and linear in rank options, and usually yields an order of magnitude more candidates than low-rank and three times as many as mixed precision.Table 6 contrasts candidate solutions for a single fully connected layer across compression approaches.Source: D. Search Space Analysis · p. 15

E. Inter-Intra Layer Metric Interpolation

Evaluating every compression configuration with inference on the calibration dataset can be costly. The section proposes linear interpolation to approximate rank-specific error with fewer inference passes.

  • Directly evaluating every configuration and layer requires separate inference on the calibration dataset, which is costly when the candidate set is large.Source: E. Inter-Intra Layer Metric… · p. 15
  • The search budgets metric evaluations per bit-width pair and interpolates between lower and upper compressed-weight options using linear factors.If a candidate set is no larger than the budget, the metric is computed for each candidate instead.Source: E. Inter-Intra Layer Metric… · p. 15
  • The interpolation is motivated by a first-order Taylor expansion of output change under weight perturbation; the prose defines the weight Jacobian and treats it as independent of the perturbation.Source: E. Inter-Intra Layer Metric… · pp. 15–16
  • Linear interpolation reduces the inference passes needed to estimate rank-specific compression error while closely approximating the actual error in the DeiT-T example.Figure 8 compares interpolated and actual per-rank error for layer blocks.1.attn.proj.Source: E. Inter-Intra Layer Metric… · pp. 15–16, Figure 8 · p. 16

F. Additional Low-Rank Ablation Results

The section reports how low-rank options affect compression results and illustrates MLoRQ’s layerwise candidate and allocation workflow. It contrasts joint optimization with sequentially dividing compression between low-rank approximation and quantization.

F. Additional Low-Rank Ablation Results

I. Sequential vs. Joint Low-Rank and Quantization

  • On DeiT-B, sequential compression performs poorly when more than 3% of weights use low-rank compression, whereas joint optimization improves performance without requiring a low-rank percentage search.Figure 9 compares the approaches at weights memory equivalent to 3-bit.Source: Figure 9 · p. 16, §I · p. 17

G. MLoRQ Compression Solutions

H. Candidates Solution Pareto Frontier Illustration

J. Algorithmic Workflow

  • The workflow’s first stage computes layerwise compression candidates by evaluating option losses and memory footprints and retaining Pareto-optimal solutions.Source: J. Algorithmic Workflow · pp. 17–18

16: Stage 2 – Inter-Layer Search

  • The inter-layer stage computes global SQNR errors for candidates, formulates an ILP, selects a compression configuration, and allocates activation precision.Source: §16 · p. 18

21: Stage 3 – Low-Rank Aware Adaptive Rounding Search

  • The final workflow stage applies quantization corrections and low-rank-aware adaptive rounding to quantized weights.Source: §21 · p. 18
Loading 2507.09616v1…