PaperlayerView exact revision on arXiv

Source-linked AI summary

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

Artem Safronov

arXiv:2608.26926v1cs.LG

TL;DR

sLLM quantization must preserve quality despite limited redundancy and memory-bandwidth-bound decoding. The paper proposes a composite metric combining normalized SQNR quality with roofline-modeled speed, and finds that FFN blocks and embeddings are promising targets with approximately 3.4% average speed-gain prediction error.

  • Problem

    sLLMs have limited redundancy, making uniform low-bit quantization risky, while autoregressive inference is constrained by the time needed to read weights from memory.

  • Method

    The method combines normalized SQNR quality scores from simulated quantization with normalized roofline-based speed scores through a tunable priority parameter α.

  • Results

    Approximately 3.4% average error across three architectures was observed for performance-gain scores, with FFN and embedding scores of S_ffn = 0.261 and S_emb = 0.236.

  • Takeaways & Limitations

    The metric supports prioritizing blocks, projections, or layers for heterogeneous quantization while balancing information preservation against performance efficiency.

  • Takeaways & Limitations

    FFN speed prediction uses one real F16 measurement as an anchor because the roofline model does not account for GPU load, scheduling, or CUDA graphs.

Abstract

from arXiv · show

Small language models (sLLMs) are nowadays hosted on devices with limited memory and computational budget. In an autoregressive setup, inference is memory-bandwidth bound: uniform quantization is often detrimental to such models, since their architecture has limited redundancies and only a few layers are not very sensitive to lower precision. We propose a composite metric that combines two orthogonal criteria: information retention (measured in terms of a normalized SQNR-based coefficient) and throughput gains (modeled using a roofline-based latency analysis). By profiling Gemma 3 1B, we find that Feed-Forward Network blocks and the embedding matrix are the most promising targets for acceleration. For each candidate, we estimate a normalized quality score based on simulated quantization and a normalized speed score based on roofline modeling with no actual execution needed. We combine the two scores in a composite priority coefficient, allowing us to tune the trade-off between speed and quality as needed. Our metric is general and can be used to prioritize individual blocks, their projection sublayers, or transformer layers as a whole. We evaluate our approach on several model architectures, showing that our estimates have at around 4% prediction error for the accelerated speedup. We find that our method generally allocates more resources to the most expressive layers compared to evolutionary search, specialized accelerators, or Shapley-value-based approaches that require expensive approximate inference. Our analytical approach makes sLLM quantization a predictable engineering task.

1 Introduction

sLLMs run under tight memory and compute constraints, while uniform quantization can damage their information-dense representations. The paper therefore proposes an analytical metric that ranks blocks by the trade-off between retained quality and inference-speed gain.

  • Motivation: sLLMs have limited redundancy, so uniform post-training quantization can substantially reduce response quality.Small models cannot discard as many weights as large models without harming usefulness.
  • Motivation: Autoregressive decoding is commonly memory-bandwidth constrained because weight transfers occupy most inference time.This makes reducing memory traffic important for practical on-device generation.
  • Method: Profiling Gemma 3 1B identifies FFN blocks and the embedding matrix as priority quantization targets because they contribute most to inference latency.The approach is intended to preserve sensitive components while allocating compression to blocks with greater hardware benefit.
  • Method: The proposed metric combines normalized SQNR-based quality and analytical roofline-based speed scores through a tunable priority parameter α.α shifts emphasis between speed and information preservation, with both equally weighted at α = 0.5.
  • Implications: The analysis can be applied without quantizing the model and extended to heterogeneous quantization across blocks or layers.This supports ranking operations by information robustness instead of applying the same precision reduction everywhere.
  • Results: Across multiple architectures, predicted speedups showed around 4% average error, while the method supported better resource allocation than several existing approaches.The comparisons include evolutionary search, specialized accelerators, and Shapley-value-based methods.

2 Formalizing the Latency Parameters of FFN and Embedding

This section develops analytical latency and sensitivity measures to prioritize quantization of model components under a speed–quality trade-off. Profiling Gemma 3 1B identifies FFN blocks and the embedding matrix as especially important targets, while SA-PTQ and layer-wise sensitivity extend the analysis beyond uniform compression.

  • 2 Formalizing the Latency Parameters of FFN and Embedding: Latency analysis should reflect memory-reading costs rather than parameter counts because sLLM autoregressive inference is memory-bandwidth constrained.The proposed cost metric estimates performance gain in seconds saved from processing model data.
  • 2 Formalizing the Latency Parameters of FFN and Embedding: Profiling Gemma 3 1B with selectively quantized blocks measured token speed, context latency, and dynamic VRAM consumption across configurations.The experiments used llama.cpp tooling, Q8_0 quantization for selected parameter groups, and F16 weights as the reference.
  • 2 Formalizing the Latency Parameters of FFN and Embedding: FFN blocks and the embedding matrix provide substantially larger speed gains from F16-to-Q8 reduction than Attention layers.The profiling results show a nonlinear relationship between bit width and inference-speed impact.
  • 2 Formalizing the Latency Parameters of FFN and Embedding: 72.3 tok/s is the short-context speed of the configuration leaving FFN and other selected components at F16, below the fully quantized alternative.Leaving the embedding unquantized also causes a substantial throughput drop.
  • 2 Formalizing the Latency Parameters of FFN and Embedding: The integrated selectivity metric combines SA-PTQ-based information analysis with analytical hardware-cost measures instead of labor-intensive latency profiling.This approach is intended to replace guesswork with computation based on SQNR and layer cost.
  • 2.1 Theoretical Basis for Layer Sensitivity Estimation: SA-PTQ models output-signal degradation through quantization and dequantization, while its step width depends on the chosen bit count.The rounding operation retains the signal components represented at the selected precision.
  • 2.1.1 Formula variant for FFN (weight quantization): Layer-wise SA-PTQ probing produces a sensitivity profile whose Q(n) values characterize information preservation and support predicting quality loss for quantized layer subsets.The FFN is treated as a major weight-bearing model-memory component, while attention is generally more fragile and embeddings are compression candidates requiring empirical verification.

3 Testing: Embedding Data

The embedding experiments examine how quantization changes token rankings and prediction confidence across individual prompts and multiple generation steps. SQNR supplies a normalized information-preservation score that can be combined with the broader metric, while layer-wise probing supports ranking components by compression tolerance.

  • Embedding ablation: Figure 2 examines token competition at one generation step by comparing the top-five token rankings for a single prompt.Prompt-level analysis also supports ranking prompts by sensitivity to reduction.
  • Embedding ablation: Top-1 matching against F16 measures the share of prompts whose winning token remains unchanged after output-layer quantization.A declining match ratio indicates prediction degradation as weight reduction increases.
  • Confidence diagnostics: The average top-1–top-2 logit gap over generation steps measures confidence robustness relative to F16.Larger gaps indicate predictions that are more robust to perturbations, including quantization.
  • Confidence diagnostics: Positive ∆gap indicates reduced separation between the winning token and its closest competitor, whereas negative values reflect stochastic quantization variation rather than improved quality.The metric is evaluated per prompt to assess vulnerability across input contexts.
  • Confidence diagnostics: SQNR evaluates distortion across the entire logit vector, complementing the local top-token gap diagnostic.Together, SQNR and ∆gap distinguish uniform distributional distortion from local shifts affecting candidate tokens.
  • SQNR normalization: Normalizing SQNR converts the distortion measure into a quantization-efficiency score for the generalized metric.The same normalization can be applied block-wise or independently to each layer.
  • SQNR normalization: 45.61 dB and 0.819 are the Q8-ch mean SQNR and normalized information-preservation score, respectively, with Q16 at 93.82 dB as reference.The normalized embedding score is later combined with a speed component in the unified metric.

4 Testing: FFN Data

The FFN experiments quantify quantization degradation through static SQNR and weight-error analyses, alongside token-by-layer heatmaps of autoregressive generation. Across configurations, degradation varies by projection, bit width, and layer, with severe losses under 4-bit quantization.

  • Layer and projection sensitivity: Q8-ch had lower SQNR for Wdown than for Wgate and Wup, while Q4-ch placed all three projections below 20 dB.The middle layers generally showed SQNR declines, with partial recovery toward the output layer.
  • Output SQNR: Q8-ch produced a mean FFN-output SQNR of 29.0 dB, whereas Q4-ch, Q8-tensor, and Q4-tensor fell below 20 dB.Q4-tensor reached around 0 dB, corresponding to complete signal destruction.
  • Weight distortion: Q8-ch showed the smallest weight-matrix error, while Q4-tensor produced an order-of-magnitude larger deviation.Wdown consistently had lower MAE than Wgate and Wup across configurations, reflecting differences in matrix dimensionality and weight distribution.
  • Autoregressive dynamics: The heatmaps showed Q8-ch degradation varying across layers, while Q4-ch produced SQNR below 10 dB across nearly the entire token-by-layer space.The dominant degradation factor was layer index, with input semantics locally amplifying the effects.
  • Normalized quality: The FFN normalized quality score was Qffn = 0.796, reflecting its higher sensitivity to compression losses.The score is subsequently used with inference-speed information to assess the information-loss versus speedup trade-off.

5 Formalizing the Latency Parameters of FFN and Embedding (Speed Component)

The section models quantization speed analytically for memory-bound embedding and FFN operations, combining roofline estimates with selective real measurements where absolute timing requires calibration. Across tested architectures, predicted speedups closely matched actual results.

  • 5 Formalizing the Latency Parameters of FFN and Embedding (Speed Component): SQNR measures distortion, whereas speed depends on bytes read from memory, so both dimensions are required for quality-versus-speed analysis.The speed component is modeled separately because SQNR contains no throughput information.
  • 5.1 Embedding: For memory-bound embedding operations, roofline modeling derives token time from memory bandwidth and weight volume without executing the model.The embedding matrix-vector multiplication has arithmetic intensity far below the hardware ridge point.
  • 5.1 Embedding: Q8 embedding quantization halves data volume, and the total token time combines transformer-block time with output-layer time.When only the embedding matrix changes, transformer-block time remains fixed while lm_head time decreases.
  • 5.1 Embedding: The embedding prediction achieved 2.6% token-time error and predicted speedup ×1.178 versus actual ×1.148, with normalized speed score Semb = 0.236.The normalized score represents the realized fraction of theoretical Q8 speedup potential.
  • 5.2 FFN: FFN absolute timing uses one measured F16 anchor because roofline modeling predicts the F16-to-Q8 difference but not system overheads.The anchor incorporates GPU load, scheduling, CUDA graphs, and other effects outside the roofline model.
  • 5.2 FFN: FFN timing is estimated from weight volume and bandwidth, with three SwiGLU projection matrices and Q8 reducing each element from 2 bytes to 1 byte.Attention, normalization, and residual operations remain in F16 and are excluded from the saving calculation.
  • 5.2 FFN: Across three models, FFN speedup prediction errors were 0.7%, 3.3%, and 6.3%, averaging 3.4%, with normalized score Sffn = 0.261.The largest deviation occurred for Qwen 2.5 1.5B and was attributed possibly to architectural and CUDA-scheduler effects.
  • 5.2 FFN: For Gemma 3 1B-it, the predicted FFN speedup was η ≈1.254 versus η = 1.183 for uniform Q8, while layer-wise heterogeneity theoretically adds about 6%.Sensitive middle layers retain Q8 while robust outer layers are quantized more aggressively; practical verification was unavailable for this architecture.

6 The Unified Metric

The unified metric combines normalized quality and speed scores through a priority parameter, enabling the weighting to shift between information preservation and performance. For Gemma 3 1B-it Q8 at α = 0.5, the resulting score was 0.497.

  • 6 The Unified Metric: The four normalized inputs are Qemb = 0.819, Qffn = 0.796, Semb = 0.236, and Sffn = 0.261.Quality scores use normalized SQNR, while speed scores represent realized theoretical speedup potential.
  • 6 The Unified Metric: The metric averages normalized speed and quality across evaluated blocks, with α ∈[0, 1] setting the priority between them.The block-level variant evaluates embedding and FFN, while the layer-wise variant evaluates all transformer layers.
  • 6 The Unified Metric: At α = 0.5, Gemma 3 1B-it’s Q8 configuration scored 0.497, comprising speed component 0.187 and quality component 0.808.This weighting balances the two normalized components.

7 Conclusion

The conclusion presents the metric as an analytical way to combine information retention with latency gains when prioritizing sLLM quantization. It identifies FFN and embedding targets, reports low speedup-prediction error, and supports use at multiple granularities.

  • 7 Conclusion: The metric combines normalized SQNR-based information retention with analytical roofline latency, without training candidate configurations.These criteria are designed to quantify quantization efficiency jointly rather than in isolation.
  • 7 Conclusion: Profiling identified FFN blocks and embedding as the largest latency contributors, while attention was less costly.These profiling results informed the choice of priority quantization targets.
  • 7 Conclusion: The speedup sanity check across three architectures produced an average error of approximately 3.4%.The conclusion presents this as evidence of practical applicability.
  • 7 Conclusion: With α = 0.5, the Gemma 3 1B-it Q8 configuration achieved score 0.497 while balancing information preservation and performance efficiency.The parameter can be adjusted according to the use case and system constraints.
Loading 2608.26926v1…