PaperlayerView exact revision on arXiv

Source-linked AI summary

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Peilin Sun, Guang Liang, Jin Tong, Jianxin Wu

arXiv:2609.34564v1cs.CV

TL;DR

Low-bit PTQ for Vision Transformers faces accuracy loss, and existing optimization-based reconstruction can overfit calibration data or rely on approximations. GLF-Q aligns penultimate features with local regularization and uses offline Hadamard transformations. It reports improved 3-bit image-classification performance, out-of-domain calibration robustness, and 8-bit GPU deployment benefits.

  • Problem

    Low-bit PTQ can substantially degrade accuracy, while logit-guided reconstruction may overfit limited calibration data and Hessian-guided approaches depend on approximations.

    Source: §1 · p. 2

  • Method

    GLF-Q combines penultimate-feature alignment through downstream full-precision layers with local reconstruction regularization, and uses offline Hadamard transformations to disperse activation outliers.

    Source: §1 · p. 2

  • Results

    GLF-Q substantially outperforms state-of-the-art methods with standard uniform quantizers under 3-bit image-classification quantization, shows out-of-domain calibration robustness, and reduces inference latency by 15% on average versus FIMA-Q under 8-bit deployment.

    Source: §4.6 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    Its 8-bit deployment results address practical ViT constraints from memory bandwidth and intensive attention on resource-limited edge devices.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Post-training quantization (PTQ) efficiently compresses Vision Transformers (ViTs) without retraining, yet suffers severe accuracy degradation at low bit-widths. Existing optimization-based PTQ methods guide block reconstruction via either soft logits or second-order Hessian proxies. Logit supervision is prone to overfitting on limited calibration data, while Hessian approximations incur structural truncation errors. To address these limitations, we propose \textbf{GLF-Q}, a novel PTQ framework guided by Global-Local Feature alignment. GLF-Q propagates quantized block outputs through downstream full-precision layers to align penultimate-layer representations under local output regularization, providing downstream feature supervision without explicitly approximating the Hessian or using a Taylor expansion. Furthermore, offline Hadamard transformations are introduced with zero runtime overhead to disperse activation outliers across channels, effectively contracting dynamic ranges and reducing quantization errors. Meanwhile, optimizing this loss via a Straight-Through Estimator (STE) achieves rapid convergence, bypassing continuous relaxation rounding formulations such as AdaRound. Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification. In addition, GLF-Q exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deployment.

1 INTRODUCTION

Low-bit PTQ for Vision Transformers can lose accuracy, while existing reconstruction signals have limitations on small calibration sets or rely on approximations. GLF-Q combines penultimate-feature alignment with local regularization and reports accuracy, robustness, and deployment benefits.

1 INTRODUCTION

  • Logit-guided reconstruction can overfit limited calibration data, while Hessian-guided approaches depend on tractable Hessian and second-order Taylor approximations.PTQ uses a small amount of unlabeled calibration data, unlike QAT, which requires full labeled data and expensive retraining.Source: §1 · pp. 1–2
  • GLF-Q combines penultimate-layer feature alignment through downstream full-precision layers with local reconstruction regularization, without explicit Hessian approximations or Taylor expansion.The loss is intended to improve generalization on limited calibration data.Source: §1 · p. 2
  • Offline Hadamard transformations spread activation outliers across channels, reducing quantization errors with zero runtime overhead.The paper says this contracts activation dynamic ranges; STE replaces continuous rounding relaxation to achieve fast convergence.Source: §1 · p. 2
  • Under practical 8-bit GPU deployment, GLF-Q achieves higher accuracy and lower inference latency, and it maintains strong accuracy under out-of-domain calibration, demonstrating practical hardware efficiency and strong out-of-domain generalizability.Source: §1 · p. 2

2 RELATED WORK

Related work distinguishes calibration-only PTQ from optimization-based block reconstruction. The review describes low-bit accuracy challenges for calibration-only methods and remaining limitations in logit- and Hessian-guided reconstruction.

2 RELATED WORK

  • QAT requires full labeled data and expensive retraining, whereas PTQ calibrates without retraining using a small set of unlabeled data.PTQ methods are divided into calibration-only and optimization-based approaches.Source: §2 · pp. 2–3
  • Calibration-only methods adjust quantization parameters during calibration, but suffer substantial low-bit accuracy loss and specialized non-uniform formats can be difficult to deploy on general hardware.Examples include RepQ-ViT, AdaLog, and UQ-ViT, which respectively use scale reparameterization, dynamic log-base tuning, and DeMax with NormQuant.Source: §2 · p. 3
  • Optimization-based PTQ reduces quantization loss through block-wise reconstruction, using approaches such as second-order error analysis, activation dropping, logit guidance, and Hessian losses.The review also covers outlier handling, MLP reconstruction, and Hessian proxy estimation.Source: §2 · p. 3
  • Logit-guided methods can overfit limited calibration data, while Hessian-guided methods rely on approximations that may become less accurate under low-bit quantization.The review identifies this as a limitation that remains despite advances in optimization-based reconstruction.Source: §2 · p. 3

3 METHOD

GLF-Q replaces logit- or Hessian-based block supervision with downstream penultimate-feature alignment and local reconstruction regularization. Its pipeline also uses offline Hadamard transformations and STE-based reconstruction.

3.1 PRELIMINARIES AND MOTIVATION

  • Logit-guided reconstruction can overfit limited calibration data, while Hessian proxies depend on curvature approximations and second-order Taylor accuracy, especially under low-bit perturbations.Source: §3.1 · pp. 3–4

3.3 GLF-Q FRAMEWORK

  • Offline orthogonal Hadamard rotations are fused into adjacent weights to distribute activation outliers across channels without runtime overhead.Source: Figure 1 · p. 4, §3.3 · p. 5

3.2 GLOBAL-LOCAL FEATURE LOSS

  • GLF aligns a quantized block’s downstream penultimate feature with the full-precision reference while local output reconstruction regularizes the block.Source: §3.2 · pp. 4–5
  • The downstream suffix stays frozen as gradients pass through it via STE; each reconstructed quantized block is then frozen before advancing.Source: §3.2 · p. 5

3.3 GLF-Q FRAMEWORK

  • Before block reconstruction, GLF-Q replaces MLP GELU with ReLU and reconstructs the MLP to reduce changes from the activation replacement.Source: §3.3 · p. 5
  • STE directly optimizes weights and quantization scales, converging within 3000 iterations rather than the tens of thousands required by continuous-relaxation methods such as AdaRound.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate GLF-Q on classification, detection, and segmentation, with ablations, out-of-domain calibration tests, and deployment-efficiency measurements. They use ImageNet and COCO data, representative ViT architectures, and uniform quantization under the described calibration settings.

4.1 EXPERIMENTAL SETUP

  • Experiments cover ImageNet classification and COCO detection and segmentation, using 1,024 and 256 unlabeled calibration images, respectively.The setup uses channel-wise uniform weight quantization, layer-wise uniform activation quantization, and 3,000 reconstruction iterations.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 EXPERIMENTS ON IMAGE CLASSIFICATION

  • By 3.30%, GLF-Q outperforms the second-best approach on ViT-S under 3-bit quantization, while maintaining robust accuracy across architectures.The classification evaluation covers ViT, DeiT, and Swin at 3 and 4 bits, using standard uniform quantization.Source: §4.2 · p. 7

4.3 EXPERIMENTS ON OBJECT DETECTION AND INSTANCE SEGMENTATION

  • GLF-Q achieves competitive COCO detection and instance-segmentation performance at 4 bits, with the best results on several model configurations.The evaluation uses Mask R-CNN and Cascade Mask R-CNN with Swin backbones.Source: §4.1 · p. 6, §4.3 · p. 7, Table 2 · p. 8

4.4 ABLATION STUDIES

  • 72.77%: GLF achieves the highest mean accuracy, exceeding DPLR-FIM, Logits + LF, and GF by 1.38%, 1.30%, and 0.98%, respectively.The comparison is under 3-bit ImageNet quantization; the full GLF-Q component combination also achieves the highest accuracy across evaluated architectures, and Hadamard improves accuracy across all seven.Source: §4.4 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

4.5 EXPERIMENTS ON OUT-OF-DOMAIN CALIBRATION

  • With SUN397 calibration, GLF-Q beats FIMA-Q on all three models at both bit-widths, with gains up to 31.97% on W3A3 DeiT-S.GLF also outperforms DPLR-FIM within the framework at both bit-widths.Source: §4.5 · p. 8, Table 5 · p. 9

4.6 ANALYSIS OF INFERENCE EFFICIENCY

  • 15%: GLF-Q reduces inference latency on average versus FIMA-Q while improving accuracy across all evaluated models under 8-bit deployment.Latency is measured on an NVIDIA H800 GPU using TensorRT.Source: §4.6 · p. 9, Table 6 · p. 9

4.7 ANALYSIS OF TRAINING EFFICIENCY

  • 2.0 to 2.6 times: GLF-Q is faster than FIMA-Q across seven architectures, and 1.6 to 2.2 times faster than LS-ViT.Under 3-bit ImageNet quantization, GLF-Q converges within 3,000 iterations using STE, while FIMA-Q and LS-ViT use AdaRound with 20,000 reconstruction iterations.Source: §4.7 · p. 9, Table 7 · p. 9

5 CONCLUSION

GLF-Q combines penultimate-layer feature alignment with local reconstruction regularization to address limitations of logit- and Hessian-guided reconstruction. The conclusion reports strong 3-bit classification results, out-of-domain calibration robustness, and 8-bit GPU speedups; the appendix also formulates its uniform quantizer.

5 CONCLUSION

  • GLF-Q combines penultimate-layer feature alignment through the full-precision downstream network with local reconstruction regularization.The loss targets calibration overfitting in logit-guided reconstruction and approximation limitations of Hessian-based objectives.Source: §5 · pp. 9–10
  • Standard uniform quantizers with GLF-Q substantially outperform state-of-the-art methods under 3-bit quantization on image classification.The conclusion also reports strong out-of-domain calibration robustness and speedups under 8-bit GPU deployment.Source: §5 · p. 10

A UNIFORM QUANTIZER FORMULATION

B COMPREHENSIVE DETAILS AND EQUIVALENCE PROOFS FOR OFFLINE HADAMARD TRANSFORMATIONS

The appendix details offline Hadamard transformations and their equivalence, including LayerNorm-to-RMSNorm conversion. Its conversion folds affine parameters and centering into adjacent layers while preserving final-normalization features.

B COMPREHENSIVE DETAILS AND EQUIVALENCE PROOFS FOR OFFLINE HADAMARD TRANSFORMATIONS

  • The appendix covers LayerNorm-to-RMSNorm conversion, offline Hadamard transformations and their equivalence, and Hadamard construction for dimensions that are not powers of two.The derivations use the row-vector convention.Source: B COMPREHENSIVE DETAILS AND… · p. 13

B.1 LAYERNORM-TO-RMSNORM CONVERSION

  • The LayerNorm affine scale and bias are folded into the subsequent linear layer, while RMSNorm retains the original epsilon.The appendix expresses LayerNorm using affine-free RMSNorm before folding the parameters.Source: §B.1 · p. 13
  • Centering is folded into the patch embedding and attention and MLP output projections to maintain centered residual streams across layers.The class token and positional embeddings are also centered; these updates preserve features after final normalization.Source: §B.1 · p. 13

B.2 RESIDUAL HADAMARD TRANSFORMATIONS

Offline orthogonal transformations are folded into model parameters to rotate residual and attention representations while preserving network outputs. GLF-Q's pipeline then reconstructs blocks against a frozen teacher using the downstream suffix and STE optimization.

B.2 RESIDUAL HADAMARD TRANSFORMATIONS

  • A shared orthogonal rotation is applied to the residual stream and propagated through RMSNorm, with the classification head absorbing its inverse to preserve logits.The corresponding parameter transformations are summarized in Table 8.Source: §B.2 · pp. 13–14, Table 8 · p. 14

B.3 HEAD-WISE QK AND VO TRANSFORMATIONS

  • Head-wise QK rotations preserve attention probabilities, while VO rotations folded into output projections preserve the multi-head self-attention output.The VO transformation retains the output-projection bias.Source: §B.3 · p. 14

B.4 RANDOMIZED HADAMARD CONSTRUCTION

  • The normalized randomized Hadamard construction supports hidden dimensions including 192, 384, and 768 without padding or truncation.For dimensions of the form 12 · 2^k, it combines a fixed H12 with H2^k using a Kronecker product.Source: §B.4 · p. 14

C OVERALL PIPELINE OF GLF-Q

  • GLF-Q initializes a frozen teacher and quantized student, then reconstructs each student block against teacher targets using the frozen downstream suffix.The pipeline uses 3000 iterations, normalizes and combines local and feature losses, and updates the block with Adam through STE.Source: C OVERALL PIPELINE OF GLF-Q · p. 15

D EFFECT OF LOSS FUNCTIONS ON GENERALIZATION

Under limited calibration data, GLF achieves higher validation accuracy than logits plus local feature regularization across seven architectures. The section also reports multi-seed accuracy coverage for GLF-Q under 3-bit and 4-bit quantization.

D EFFECT OF LOSS FUNCTIONS ON GENERALIZATION

  • With nearly identical mean calibration accuracy, GLF yields higher validation accuracy across all seven architectures than Logits + LF.The compared variants use the same 3-bit setting and differ only in loss function.Source: D EFFECT OF LOSS FUNCTIONS ON… · p. 15
  • GLF improves ImageNet validation top-1 accuracy by 1.30% on average across seven architectures, narrowing the calibration–validation gap from 17.23% to 15.99%.Mean calibration accuracy is nearly identical: 88.76% for GLF and 88.70% for Logits + LF.Source: Table 9 · p. 15, D EFFECT OF LOSS FUNCTIONS ON… · pp. 15–16

E STABILITY ACROSS RANDOM SEEDS

F ADDITIONAL ABLATION STUDIES

Under 3-bit ImageNet quantization, mean top-1 accuracy across seven architectures varies little across the evaluated regularization coefficients.

F ADDITIONAL ABLATION STUDIES

G IMPLEMENTATION DETAILS

The implementation details describe block-wise MLP reconstruction and feature-supervised reconstruction for object detection and instance segmentation.

G.1 MLP RECONSTRUCTION

  • MLP reconstruction trains a ReLU student against cached GELU-teacher inputs and outputs, while keeping their coordinate transformations identical.The process compensates for error from replacing GELU with ReLU.Source: §G.1 · p. 17
  • The student MLP objective uses upper clipping at a per-block threshold set to the 99th percentile of positive teacher FC2-input activations.N and C denote output-token and channel counts, respectively.Source: §G.1 · p. 17
  • Each MLP is optimized for 20,000 Adam iterations, updating only FC1 and FC2 weights and biases with an independent cosine learning-rate schedule.The learning rate decays from 4 × 10−5 to zero; other parameters remain frozen.Source: §G.1 · p. 17

G.2 OBJECT DETECTION AND SEGMENTATION

  • Object-detection and segmentation supervision uses intermediate bbox and mask RoI features, with up to 1000 and 128 student RPN proposals per image, respectively.Teacher and student paths use identical RoI coordinates, and proposals and teacher reference features are cached.Source: §G.2 · p. 17
  • Backbone reconstruction combines normalized local, bbox RoI, and mask RoI MSE losses; FPN uses local MSE, while RPN and RoI heads use calibration-only quantization.Backbone loss weights are 2, 0.5, and 0.5, respectively; FPN local MSE has weight 2.Source: §G.2 · p. 17

G.3 TENSORRT DEPLOYMENT AND LATENCY MEASUREMENT

TensorRT deployment uses a common quantization scheme for representations incompatible with its standard INT8 path, and latency is measured under specified GPU-engine conditions.

G.3 TENSORRT DEPLOYMENT AND LATENCY MEASUREMENT

  • Incompatible asymmetric quantization representations are exported in floating-point form and requantized using a common scheme for TensorRT’s standard INT8 path.Weights use symmetric absmax quantization per output channel, and activations use it per tensor.Source: §G.3 · p. 17
  • Latency is measured on an NVIDIA H800 80GB GPU with TensorRT 10.9 at 224 × 224 and batch size 64, reporting mean batch latency over 200 runs after 50 warm-ups.Timing covers GPU engine execution only; data loading, preprocessing, and transfers are excluded, with FP32/TF32 fallback allowed.Source: §G.3 · p. 17

Related papers

Loading 2609.34564v1…