Source-linked AI summary
QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncation
Lin-Fa Lee, Yi-Yu Chang, Kuo-Hei Yeh
TL;DR
Under W4A4 quantization, ill-conditioned activation structures can make closed-form compensation solvers fail numerically, causing goodness-of-fit gates to discard recoverable layers. QuaSAR uses a parameter-free truncated pseudoinverse to remove collapsed directions, reaching 81.42% top-1 accuracy on ViT-B and 80.26% at 54.7 MB after compression.
Problem
Existing W4A4 compensation methods lack a diagnosis and remedy for numerical collapse caused by rank-deficient activations, which can falsely label recoverable layers as uncompensable.
Method
QuaSAR replaces direct inversion with a parameter-free truncated pseudoinverse that removes collapsed directions while preserving the remaining compensable subspace.
Results
81.42% top-1 accuracy is achieved on ViT-B/W4A4 training-free, while compressed compensation retains 80.26% accuracy at 54.7 MB.
Takeaways & Limitations
Stable numerical compensation can recover layers discarded by standard gates and support a deployable accuracy–size trade-off under W4A4 quantization.
Takeaways & Limitations
Applying the complete layer-wise framework to CNNs remains open because module-level compensation does not yet stably outperform the block-level baseline.
Abstract
from arXiv · showhide
Recent training-free post-training quantization methods restore model accuracy through closed-form residual compensation. To constrain additional model storage overhead, several existing methods gate layer selection by goodness-of-fit, retaining only those layers whose compensation yields a positive residual fit score and discarding the rest. In this paper, we show that, under the low-bit W4A4 setting, this gating mechanism fails to distinguish poorly predictable quantization error from numerical solver failure. Rank-deficient input activations yield severely ill-conditioned or numerically singular Gram matrices, causing the closed-form solver to become unstable and produce spuriously negative fit scores. Consequently, existing goodness-of-fit gates misclassify affected layers as uncompensable and discard them. Many of these discarded layers can nevertheless provide substantial error recovery when their compensation is computed using a numerically stable solver. To address this problem, we propose a parameter-free truncated pseudoinverse solver which removes collapsed directions prior to inversion. On ViT-B with the W4A4 setting, our training-free method achieves 81.42\% top-1 accuracy, outperforming prior post-training methods and fine-tuning-based baselines. Combined with joint low-rank and quantization compression, the proposed method reaches a deployable operating point of 80.26\% accuracy at 54.7 MB, providing a well-balanced trade-off between model size and accuracy.
1 Introduction
The introduction frames training-free post-training quantization as a lightweight alternative whose closed-form compensation can fail on rank-deficient activations, especially in sensitive W4A4 vision-model settings. QuaSAR addresses this overlooked failure mechanism while targeting a better accuracy–model-size trade-off.
- Background: Quantization supports model compression and deployment on resource-constrained devices, with QAT requiring full-dataset retraining and PTQ using only a small calibration set.The introduction distinguishes quantization-aware training from post-training quantization by their data and training requirements.
- Closed-form compensation: Training-free PTQ methods model quantization error as a linear function of input, derive closed-form compensation, and add corrections to quantized outputs.These methods can recover accuracy close to the original full-precision model, but their compensation requires inverting an activation covariance matrix that is often ill-conditioned.
- Problem diagnosis: Existing stability techniques such as GPTQ dampening and ridge regularization preemptively circumvent ill conditioning without explaining compensation failure or erroneous discarding of compensable positions.The introduction identifies a gap between numerical stabilization and diagnosing why compensation fails.
- W4A4 setting: W4A4 vision-model quantization is more accuracy-sensitive than weight-only quantization, while high-dimensional activations make low-rank compensation particularly challenging.Prior closed-form low-rank compensation work has mainly focused on weight-only quantization for large language models, whereas efficient vision inference often quantizes weights and activations.
- Accuracy–size trade-off: Existing methods lack an operating point balancing accuracy and model size: QwT prioritizes accuracy with larger models, whereas QwT-v2 reduces size through static diagonal compensation but sacrifices capability.The paper positions its framework as addressing this unresolved trade-off under W4A4.
- Core contribution: The introduction identifies rank deficiency, amplified by activation quantization, as the trigger for numerical collapse that misjudges valuable layers as uncompensable and discards them.This diagnosis motivates the paper’s training-free layer-wise compensation framework and distinguishes rank deficiency from ill conditioning alone.
2 Related Work
Related work develops training-free quantization and compensation methods for deploying low-precision models, particularly addressing the severe degradation of direct PTQ on Vision Transformers. Approaches target activation distributions, reconstruct quantization parameters, exploit low-rank error structure, or directly compensate residual discrepancies.
- Post-training quantization: PTQ avoids QAT retraining for edge deployment, but direct application to ViTs suffers severe degradation from inter-channel activation variance and heavy-tailed attention maps.These effects arise in post-LayerNorm activations and post-Softmax attention maps.
- Activation-aware PTQ: RepQ-ViT decouples calibration-time and hardware-friendly inference quantization grids, while IGQ-ViT groups activation channels by instance to isolate token-dependent outliers.Both frameworks specifically target problematic activation distributions in ViTs.
- Low-rank compensation: Low-rank compensation methods augment quantized backbones with high-precision components, including LoftQ’s joint initialization and QERA’s closed-form output-discrepancy minimization.These methods exploit or approximate the low-rank structure of quantization errors.
- Residual compensation: Structural compensation methods directly correct activation discrepancies; QwT adds lightweight parallel linear layers and uses closed-form least-squares regression to predict and add back residual quantization error.This avoids the overhead of parallel high-precision branches while recovering substantial accuracy without backpropagation.
3 Proposed Method
QuaSAR introduces layer-wise residual compensation with sequential deployment, a numerically stable truncated-pseudoinverse solver for W4A4 collapse, and dual-axis compression of the compensator. The method targets rank-deficient activations that cause unstable closed-form solutions and erroneous layer rejection while reducing deployment overhead.
- Layer-Wise Compensation: Independent compensators attach to each of four linear layers, improving accuracy by +3.1% over block-wise compensation.Each compensated layer is replaced before capturing the next layer’s true input, maintaining consistency with deployment.
- Layer-Wise Compensation: Sequential compensation immediately reinserts each corrected layer before solving the next, preventing mismatch between captured inputs and deployment inputs.Compensating one layer changes downstream inputs, so all layers cannot be solved from shared pre-compensation activations.
- Numerically Stable Solving: Under W4A4, rank-deficient activations make Gram matrices numerically singular, causing exploding ∥Wc∥ and sharply negative R2 that existing R2 > 0 gates misclassify as uncompensable.ViT-B condition numbers can reach 10^7–10^9, but large κ alone does not predict collapse.
- Numerically Stable Solving: Truncated pseudoinversion retains eigen-directions above τ = rcond · λmax and discards collapsed directions, eliminating numerical explosion while preserving the R2 ≥0 guarantee.The ignored directions are treated as unreliable, allowing high-value layers discarded by existing methods to be compensated.
- Numerically Stable Solving: 81.42% accuracy from truncated pseudoinverse and 81.64% from ridge regularization show that the improvement comes from stability rather than shrinkage.The truncated pseudoinverse sets eigenvalues below the threshold to zero without shrinking retained directions.
- Dual-Axis Compression: Dual-axis compression combines activation-aware low-rank decomposition with channel-wise symmetric int8/int4 quantization of factors A and B.Whitening uses the same truncated eigendecomposition threshold as the stable solver, while the bias bc remains full precision.
4 Experiments
Under W4A4 quantization, QuaSAR is evaluated across four architectures and achieves 81.42% Top-1 accuracy on ViT-B through numerically stable compensation. Joint compression reaches 80.26% accuracy at 54.7 MB, while experiments examine solver stability and rank allocation.
- Main results: 81.42% ± 0.06 Top 1 accuracy is achieved training free on ViT-B/W4A4, outperforming QwT (76.3%), QwT* (78.5%), QwT-v2 (75.6%), and IGQ-ViT (79.3%).Accuracy improves from 76.43% with block wise compensation to 79.52% after pushing compensation to the linear layer, then 81.42% with a numerically stable solution.
- Solver stability: Direct inversion collapses in 3 out of 5 seeds, reducing collapsed-seed accuracy to approximately 0.1%, whereas the truncated pseudo inverse remains stable across all seeds.The direct solver’s 79.52% average is attributed to uncollapsed seeds, while the truncated pseudo inverse achieves 81.42% ± 0.06.
- Compensation ablation: R2 > 0 gating discards layers whose direct compensation is numerically misjudged, including b0.fc2, whose removal causes a 1.05% accuracy drop.The ablation identifies b0.fc2 as the most valuable among all 48 compensators despite its erroneous discard decision.
- Cross-architecture evaluation: Ill conditioned Gram matrices occur across ViT-B, DeiT-T, Swin-T, and ResNet-50, with ViT-B reaching a maximum condition number of 2.3 × 10^9.The resulting failures include silent least-squares collapse on ViT-B and high variance with individual-seed degradation on Swin-T.
- Accuracy–size trade-off: 80.26% ± 0.11 accuracy at 54.7MB provides the primary training-free accuracy–size operating point and leads QwT* by 1.76% at 4.4MB smaller size.Under the same 54.7MB constraint, uniform rank allocation achieves 80.376%, while position-wise allocation is not superior.
5 Conclusion and Future Work
The paper identifies numerical collapse in activation-quantized Gram matrices as the cause of false uncompensable-layer gating and replaces direct inversion with a truncated pseudoinverse. Future work includes extending the framework to CNNs and lower-bit settings.
- Conclusion: Activation quantization can numerically collapse compensation Gram matrices, causing the standard R2 > 0 gate to discard layers with high compensation value.The paper attributes this failure to numerical instability rather than genuinely unpredictable quantization error.
- Conclusion: A truncated pseudoinverse replaces direct inversion to salvage layers falsely rejected by the standard gate.The stable solver is reported to rescue compensation on ResNet-50.
- Future Work: Applying the complete layer-wise compensation framework to CNNs remains open because module-level compensation does not yet stably outperform the block-level baseline.The paper attributes this limitation to convolutional layers’ error propagation structure.
- Future Work: Lower bit widths worsen ill conditioning, making settings such as W3A3 an open direction for future investigation.The supplied passage identifies decreasing bit width as a source of worsening numerical conditioning.