Source-linked AI summary
RAMP: Robust Adaptive Mixed-Precision Quantization for Edge CPU Vision Models
David Población-Criado, Dario Garcia-Gasulla, Eduardo Quinones
TL;DR
Edge CPU deployment faces resource constraints, and sensitivity metrics used to guide mixed-precision quantization can fail on modern architectures. The paper evaluates these proxies and uses JSD with K-Means clustering to allocate layer precision. Across ARM64 platforms, this approach avoids catastrophic accuracy collapse while achieving near-lossless accuracy and inference speed-ups.
Problem
Sensitivity proxies can fail to predict quantization degradation in modern architectures, complicating layer-wise precision allocation under edge CPU constraints.
Source: §1 · pp. 1–2Method
The paper systematically evaluates sensitivity proxies and proposes solver-free RAMP, which applies K-Means clustering to JSD to generate adaptive quantization policies.
Source: §1 · pp. 2–3Results
Across four architectures and two ARM64 platforms, JSD with 1D K-Means consistently prevents catastrophic accuracy collapse and achieves near-lossless accuracy with substantial inference speed-ups.
Source: §7.1 · p. 10Takeaways & Limitations
Maintaining contiguous integer subgraphs matters more to end-to-end latency than optimizing isolated layer execution times, and RAMP requires neither retraining nor architecture-specific tuning.
Source: §7.1 · p. 10The pipeline was evaluated exclusively on image classification benchmarks, and physical speed-ups depend on the execution engine.
Source: §7.2 · p. 11
Abstract
from arXiv · showhide
Deploying deep learning models on edge CPUs is bottlenecked by computational and memory constraints. Mixed-precision quantization promises to reduce inference latency while preserving accuracy. However, quantization affects different layer types in inconsistent ways, so identifying where accuracy loss is minimized and latency reduction is maximized is critical, as the effect accumulates over a full deployment into substantial savings or unacceptable task degradation. Such identification relies on sensitivity metrics, proxies that estimate layer-wise degradation without evaluating the task accuracy of every candidate policy. Nevertheless, widely used metrics fail systematically on modern architectures. We present a systematic empirical study of 13 sensitivity metrics for layer-wise INT8 quantization across four distinctly different neural networks, and validate the resulting policies on two ARM64 platforms. Gradient-based sensitivity methods fail on 4 out of 8 model-hardware configurations and weight-based statistics on 2. In contrast, the Jensen-Shannon Divergence achieves zero catastrophic failures, reliably isolating the layers that cannot be safely quantized. A sensitivity metric alone does not define a policy, and the fixed thresholds typically used for that step are fragile over the highly skewed distributions of modern architectures. We address this with K-Means clustering, achieving near-lossless accuracy and a mean speed-up of $1.81\times$ over the full-precision model. Finally, we reveal that excluding from quantization the layers whose speed-up is negligible, regardless of their sensitivity, can be counterproductive, as it induces computational graph fragmentation and disables operator fusion. Our results yield concrete allocation policies for practitioners and researchers deploying quantized vision models on heterogeneous edge CPUs, without GPU access or gradient computation.
1 Introduction
Edge CPUs impose latency, energy, and memory constraints on deep learning deployment, while quantization offers acceleration but can damage accuracy unevenly across layers. The paper evaluates sensitivity metrics and introduces adaptive clustering to address accuracy and deployment costs.
1 Introduction
- Edge deployment must meet latency, energy, and memory constraints across architectures ranging from CNNs to ViTs.The paper notes that models designed for high-end servers must be adapted for resource-constrained devices.Source: §1 · p. 1
- Uniform low-bitwidth quantization can cause catastrophic accuracy loss, while layer types differ substantially in their tolerance to precision loss.The paper focuses on quantization because it can accelerate edge hardware without access to the original training pipeline.Source: §1 · p. 2
- Under one fixed deployment configuration, INT8 speed-up spans from 5× acceleration to net slowdown across CPUs.Figure 1 covers four vision models and three CPUs, using ONNX Runtime's default CPU provider, static PTQ, and batch size 1; its dashed line marks 1×.Source: §1 · p. 2, Figure 1 · p. 2
- Jensen-Shannon Divergence (JSD) is a better quantization-degradation proxy than gradient-based metrics and prevents accuracy collapse across diverse architectures.Source: §1 · p. 2
- Strict layer-level constraints can fragment the computational graph and disable operator fusion, outweighing individual operator acceleration.Source: §1 · p. 3
- RAMP applies 1D K-Means clustering to the JSD landscape to generate adaptive, near-lossless quantization policies.Source: §1 · p. 3
2 Background and Related Work
Mixed-precision quantization allocates different precisions across layers, using sensitivity proxies to guide allocation. The section contrasts search-based and criterion-based methods and motivates adaptive policies for heterogeneous CPUs.
2.1 Mixed-Precision Quantization
- Mixed-precision quantization assigns different precisions to layers, channels, or tensors, seeking to preserve sensitive components while compressing less consequential ones.With L layers and candidate precision set B, the search space grows as O(|B|^L).Source: §2.1 · p. 3
- Search-based methods explore candidate allocations at substantial computational cost, whereas criterion-based methods use sensitivity proxies to guide allocation more efficiently.The section describes search-based methods as potentially requiring days or weeks and access to the training pipeline and full dataset.Source: §2.1 · p. 3
- Fixed percentile thresholds can fail on modern networks' skewed sensitivity distributions, motivating more adaptive grouping mechanisms.Sensitivity proxies include curvature-based, information-theoretic, and signal-distortion measures.Source: §2.1 · p. 3
2.2 Target Hardware
- CPU deployment is under-explored, and heterogeneous instruction implementations and cache bottlenecks complicate latency estimation.The study centers its empirical scope on ARM64, where INT8 is uniformly available at the ISA level.Source: §2.2 · pp. 3–4
3 Problem Formulation
The formulation treats mixed-precision allocation as minimizing task loss under a resource constraint, using layer sensitivity as a proxy because evaluating every candidate is prohibitive. It separates the sensitivity metric from the method that allocates precision.
3 Problem Formulation
- With coupled weight and activation precision, each layer receives one bit-width, yielding |B|^L configurations, or 2^L when there are two choices.Source: §3 · p. 4
- The objective is to minimize task loss subject to a resource constraint, but exhaustive evaluation is computationally prohibitive, so layer sensitivity serves as a proxy.The sensitivity penalty compares full-precision and quantized representations or outputs for a layer.Source: §3 · p. 4
- The optimization is framed as discrete subset selection under a cost constraint, while the solver-free approach uses clustering instead of explicit resource restrictions.A complete criterion-based method combines a sensitivity metric with a search method for bit-width allocation.Source: §3 · p. 4
4 Mixed-Precision Quantization
The section evaluates sensitivity proxies and develops an adaptive policy for layer-wise mixed-precision quantization. It finds that JSD identifies vulnerable layers and K-Means thresholding avoids the accuracy collapse associated with fixed percentiles.
4.1 Experimental Setup
- The sensitivity study covers ResNet-18, EfficientNet-B0, ConvNeXt-Tiny, and TinyViT on the synthetic SuSy deepfake-detection dataset.The models span classical CNNs to vision transformers; the dataset tests sensitivity to quantization-induced distortion of high-frequency artifacts.Source: §4.1 · p. 4
4.2 Sensitivity Analysis
- One-at-a-time testing quantizes each layer independently against an FP32 baseline, while Recall@Top-N% measures agreement on the most accuracy-sensitive layers.This ranking does not model inter-layer dependencies or joint-quantization error accumulation.Source: §4.2 · p. 5
- Metric correlations with accuracy degradation vary by architecture and are not statistically significant for TinyViT, so proxies are assessed by top-layer retrieval.Information-theoretic and distance metrics are highly inter-correlated (r > 0.9).Source: §4.2 · p. 5
- Output-space JSD and KL average above 0.76 mean recall; on TinyViT Top-10%, HAWQ-V2 scores 0.333 versus JSD’s 0.833.JSD is selected as the framework’s proxy; cosine dissimilarity has 0.726 mean recall, while SNR and estimated SQNR perform poorly.Source: §4.2 · pp. 5–6, Table 1 · p. 6
4.3 Distribution-Adaptive Thresholding
- K-Means partitions layer sensitivity scores by minimizing within-cluster variance, then maps lower-sensitivity clusters to INT8 and extreme outliers to FP32.This distribution-adaptive policy avoids imposing an arbitrary layer quota per group.Source: §4.3 · p. 6
- Fixed percentiles trigger accuracy collapse by quantizing topological outliers, whereas K-Means preserves model semantics across the remaining evaluated models.On TinyViT, K-Means isolates low-sensitivity layers at its first threshold; the percentile baseline does not do so until the 90th percentile.Source: §4.3 · pp. 6–7, Figure 2 · p. 7
5 Hardware Impact
INT8 speed-ups on CPUs vary by architecture, backend, and layer type, so lower precision does not guarantee lower latency. The measurements motivate profiling hardware behavior when designing layer-wise quantization policies.
5 Hardware Impact
- On Apple M1, ConvNeXt-Tiny convolutional layers reach 6× to 15× speed-up, while on Cortex-A76 only a minority exceed 6×.The caption reports convolutional layers reaching up to 15×, while normalization and element-wise operations show minimal gains.Source: §5 · p. 7, Figure 3 · p. 7
- With the default ONNX Runtime CPU provider, x86-64 INT8 sometimes worsens latency versus FP32; OpenVINO restores positive speed-ups below 1.4×.The evaluated CPUs are Apple M1 and Raspberry Pi 5 (ARM64), plus Intel Core i9-12900KF (x86-64).Source: §5 · pp. 7–8
- INT8 benefits depend on hardware and workload; conversion overhead can exceed computation savings for low-arithmetic-intensity layers such as small depthwise convolutions.Cache capacity and memory bandwidth also affect weight-loading latency, especially for small tensors.Source: §5 · pp. 7–8
- Under the additive layer-cost model, profiling suggests layers with negative or negligible speed-up may be poor quantization candidates regardless of sensitivity.The paper treats this implication as a hypothesis and evaluates it as an ablation.Source: §5 · p. 8
6 The RAMP Method
RAMP combines JSD sensitivity profiling, K-Means thresholding, and Pareto-front policy selection for mixed-precision quantization. JSD-driven policies avoid accuracy collapse, while latency filtering can hurt end-to-end performance through graph fragmentation.
6 The RAMP Method
- RAMP profiles layer sensitivity with JSD, uses K-Means to find sensitivity breakpoints, and generates candidate policies from ordered cluster centroids.The operational policy is chosen from the candidates on the accuracy–latency Pareto front using the knee-point method.Source: §6 · p. 8
6.1 End-to-End Validation
- JSD-driven thresholding avoids performance degradation across evaluated models and platforms; TinyViT accuracy is 97.3% on Apple M1 and 97.4% on Raspberry Pi 5 versus 97.5% for FP32, while ConvNeXt-Tiny latency falls from 168 ms to 95 ms on Raspberry Pi 5.Table 2 reports end-to-end accuracy and latency.Source: §6.1 · p. 9, Table 2 · p. 9
6.2 Graph Fragmentation
- Latency filtering leaves accuracy-collapse failures at zero of eight configurations but worsens latency in four, improves it in two, and leaves it unchanged in two.On Raspberry Pi 5 it worsens latency for three of four models, including ConvNeXt-Tiny at identical accuracy; Apple M1 results are neutral.Source: §6.2 · p. 10
- FP32 fallbacks fragment contiguous INT8 subgraphs, add Q/DQ nodes, and prevent fusion; depthwise-convolution latency rises from 7.7 ms to 16.5 ms.The fragmentation cost can exceed the latency saved by protecting individual layers.Source: §6.2 · p. 10, Figure 4 · p. 10
- Because latency penalties depend on neighboring layers, per-layer additive cost tables cannot model fragmentation; RAMP therefore omits latency profiling and transfers its JSD policy across both platforms.The reported setting is ARM64 platforms with native INT8 SIMD support.Source: §6.2 · p. 10
7 Conclusion and Future Work
The paper concludes that JSD with 1D K-Means supports near-lossless quantization with substantial speed-ups across ARM64 platforms, while graph fragmentation makes contiguous integer subgraphs important to end-to-end latency. Its findings are bounded by calibration-data representativeness, evaluated task domains, hardware backends, and one-at-a-time profiling.
7.1 Conclusion
- JSD with 1D K-Means prevents catastrophic accuracy collapse and supports near-lossless accuracy with substantial speed-ups across diverse ARM64 platforms.The sensitivity study covers four model architectures and two ARM64 CPU platforms.Source: §7.1 · p. 10
- Maintaining contiguous integer subgraphs affects end-to-end latency more than optimizing isolated layer execution, challenging additive per-layer latency models.The conclusion also notes that the pipeline requires no retraining or architecture-specific tuning.Source: §7.1 · p. 10
7.2 Future Work
- Results cover image classification and binary FP32/INT8 policies; JSD depends on representative calibration data, and one-at-a-time profiling may miss layer interactions.The authors identify dense prediction, NLP, calibration-free thresholding, and block-wise profiling as areas requiring further validation or work.Source: §7.2 · p. 11