PaperlayerView exact revision on arXiv

Source-linked AI summary

SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

Ziyi Wang, Nan Jiang, Guang Lin, Qifan Song

arXiv:2510.08999v2cs.LGcs.AI

TL;DR

DNN compression must reduce computational and memory requirements while preserving model performance, yet separate pruning and quantization can limit achievable compression. SQS jointly learns sparse and quantized weights through Bayesian variational learning using spike-and-slab and GMM components. Across ResNet, BERT-base, Llama3.2, and Qwen2.5, it reports higher compression rates with comparable performance degradation than existing baselines, while its theoretical analysis has narrower scope than its experiments.

  • Problem

    Existing compression methods often apply pruning or low-bit quantization separately, making high compression with acceptable performance an open problem for resource-constrained deployment.

  • Method

    SQS jointly learns pruning and low-bit quantization through Bayesian variational learning with a spike-and-slab prior and GMM-based quantized weight distribution.

  • Results

    Across ResNet, BERT-base, Llama3.2, and Qwen2.5, SQS achieves higher compression rates with comparable or smaller accuracy degradation than existing baselines.

  • Takeaways & Limitations

    SQS supports efficient deployment in resource-constrained environments while maintaining competitive accuracy across the evaluated model benchmarks.

  • Takeaways & Limitations

    LLM results characterize compression of task-adapted models, while the theoretical analysis is restricted to regression problems with fully connected neural networks.

Abstract

from arXiv · show

Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (\method), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to induce sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. Due to the intractability of the objective involving spike-and-slab priors with GMMs, we derive an efficient approximation that facilitates effective compression with minimal accuracy loss. In theory, we provide a consistent result for our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3.2, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops. Project page: https://comeusr.github.io/SQS_Webpage.

1. Introduction

SQS addresses the challenge of compressing DNNs without unacceptable performance loss by jointly learning pruning and low-bit quantization in one variational framework. Experiments across convolutional, transformer, and language models report higher compression rates with comparable or smaller performance degradation than existing methods.

  • DNN compression reduces model size and computational complexity for deployment on resource-constrained devices, but preserving predictive accuracy remains challenging.
  • SQS jointly learns which weights to remove and how to quantize the remaining weights using a spike-and-GMM variational distribution.The spike component promotes sparsity, while the GMM models a quantized weight distribution.
  • SQS achieves higher compression rates than existing baselines with comparable or smaller accuracy degradation across ResNet, BERT-base, Llama3.2, and Qwen2.5.
  • At the same compression rate, SQS achieves the smallest accuracy drop or F1 score drop among evaluated approaches, particularly at 2-bit and 4-bit precision.
  • Ablations support spike-and-slab sparsity, Bayesian averaging at inference, and outlier-aware windowing as effective components of SQS.The outlier-aware strategy better preserves informative weight outliers than uniform windowing.

2. Preliminaries

The preliminaries define low-bit quantization and Bayesian variational learning, then motivate SQS's spike-and-GMM posterior for sparse, quantized weights. The framework addresses the difficulty of achieving high compression while limiting performance degradation.

  • Low-bit quantization maps full-precision weights to a finite set of discrete values to reduce storage and computation while preserving essential information.It is represented as Q: ℝ→Q = {μ1, . . . , μK}.
  • A GMM approximates the quantized weight distribution and bridges continuous weights with a multinomial distribution over the quantization set.
  • The GMM-based compression method cannot achieve high compression with a small performance drop because it does not efficiently encourage sparsity during training.
  • Variational learning approximates an intractable Bayesian posterior by optimizing within a variational family through the negative Evidence Lower Bound.The likelihood term fits the data, while the regularization term keeps the variational distribution close to the prior.
  • SQS uses a spike-and-GMM variational family to approximate a sparse and quantized posterior, with a spike-and-slab prior promoting sparsity.

3. Methodology

SQS jointly learns sparsity and low-bit quantization through a spike-and-GMM variational framework, with approximate training and Bayesian inference. Its design includes deterministic sparsity control, layer-wise outlier-aware quantization, and theoretical and empirical validation.

  • Variational formulation: SQS approximates sparse and quantized weights with a spike-and-GMM variational posterior.The spike component promotes pruning, while the GMM models quantized weight values.
  • Variational formulation: The spike-and-slab prior represents retained weights with a slab and pruned weights with a point mass at zero.The prior retention probability is λ, while 1−λ is the prior pruning probability.
  • Training procedure: Because the ELBO contains an intractable KL divergence, SQS replaces it with an approximate objective using coordinate-wise posterior means and upper bounds.The likelihood is evaluated using the complete vector of approximate means, with additional terms bounded through Lemma 3.
  • Inference procedure: SQS deterministically prunes weights below the non-zero quantile of retention scores, providing exact control over the sparsity level.Weights above the threshold remain unchanged, unlike stochastic pruning schemes that do not guarantee a fixed sparsity rate.
  • Inference procedure: Bayesian averaging samples sparse quantized weights and averages predictions, yielding smaller accuracy degradation than greedy inference.The experiments use M=4 posterior samples by default.
  • Theory and scope: SQS combines pruning and quantization in a Bayesian joint optimization space and is theoretically analyzed for convergence under mild conditions.The theory focuses on fully connected networks, while experiments evaluate ResNets, BERT-based models, and LLMs.
  • Quantization strategy: Outlier-aware windowing partitions layer weights into four groups and preserves extreme values during quantization.The method uses tail-focused windows based on an interquartile-range rule and applies GMM quantization within windows.

4. Related Work

Prior work studies pruning, low-bit quantization, and joint compression through diverse optimization and Bayesian formulations. SQS is positioned among methods that combine sparsity and precision while emphasizing a unified spike-and-GMM Bayesian treatment.

  • Weight pruning: Weight pruning removes selected parameters using approaches based on second-order information, regularization, or subnetworks.Recent work also combines structured pruning with targeted retraining for large language models.
  • Low-bit quantization: Low-bit quantization includes discontinuous grid projection and continuous-mapping methods using variational learning or Markov Chain Monte Carlo.The two paradigms differ in how they represent or optimize discrete weight distributions.
  • Joint compression: Joint pruning and quantization methods include Bayesian priors or posteriors, differentiable gates, saliency criteria, and policy search.Bayesian Bits unifies the tasks by gating residual terms, with pruning represented as the 0-bit case.
  • Large language model compression: Recent large language model compression methods address outliers, adaptive quantization, and domain-aware pruning through rotations, affine transformations, or LoRA guidance.These approaches pursue complementary strategies for preserving difficult weights or reducing model size.

5. Experiments

Across ResNet, BERT-base, Llama3.2, and Qwen2.5 experiments, SQS achieves higher compression with comparable or smaller accuracy or F1 drops than baselines. Ablations attribute these results to spike-and-slab sparsity, Bayesian averaging, and outlier-aware windowing.

  • Benchmark results: SQS achieves higher compression rates with comparable or smaller performance drops across the evaluated model families.The experiments cover ResNet, BERT-base, Llama3.2, and Qwen2.5, with performance measured by accuracy or F1 score drop.
  • Benchmark results: SQS achieves the highest compression rate and lowest F1 score drop among compared methods on BERT-base.The evaluation uses BERT-base on SQuAD v1.1.
  • Prior ablation: The spike-and-slab prior consistently outperforms a Gaussian prior across sparsity levels, especially at high sparsity.The comparison compresses ResNet-18 with K=16 components and evaluates CIFAR-100 Top-1 accuracy.
  • Windowing ablation: At 6 bits and 50% nonzero weights, outlier-aware windowing reduces Qwen2.5-0.5B’s accuracy drop from 5.40 to 2.46 percentage points.This is a 2.94-percentage-point difference at the same bit width and nonzero rate.
  • Inference ablation: Bayesian averaging consistently produces smaller accuracy drops than greedy inference under matched Gaussian-component counts.With posterior averaging, a single sample gives a 3.76% Top-1 drop, while 50 samples reduce it to 2.63%; five samples already give 2.86%.

6. Conclusion

SQS integrates pruning and quantization through variational learning, and experiments report higher compression rates with competitive accuracy across several model families. The paper also identifies deployment, safety, task-adaptation, and theoretical-scope considerations.

  • 6. Conclusion: SQS combines pruning and quantization in one variational optimization process rather than applying them sequentially.The method uses an efficient approximation to make optimization scalable.
  • 6. Conclusion: SQS consistently outperforms existing baselines in compression rate while maintaining competitive accuracy across ResNets, BERT-base, Llama3.2, and Qwen2.5.
  • 6. Conclusion: Deployment should measure end-to-end latency, memory use, energy consumption, and accuracy on the target hardware and workload.Bayesian averaging can improve robustness but may increase inference latency, while compression requires additional post-pretraining optimization.
  • 6. Conclusion: SQS inherits the base model’s safety risks, so compressed models require post-compression evaluation for robustness, fairness, privacy leakage, and failure modes.
  • 6. Conclusion: LLM results characterize compression of task-adapted models, not preservation of general-purpose capabilities, and the theory does not directly cover transformers or classification.Llama3.2 and Qwen2.5 were fine-tuned on SST-2 before compression; omitting adaptation caused substantial degradation.

A. Derivation of Approximate Objective

The approximate-objective derivation addresses intractable likelihood and KL terms arising from spike-and-GMM variational distributions. It combines posterior-mean, KL upper-bound, and dominant-component approximations into a tractable training objective.

  • A. Derivation of Approximate Objective: The derivation targets an ELBO whose expected log-likelihood and KL terms are expensive or intractable under the spike-and-GMM distribution.The KL divergence between the variational distribution and spike-and-slab prior lacks a closed form.
  • A. Derivation of Approximate Objective: The expected log-likelihood is approximated by evaluating the likelihood at the posterior mean parameter vector.
  • A. Derivation of Approximate Objective: The spike-and-slab KL divergence is upper-bounded by matching the mixture components through a KL inequality.The component structure includes the Dirac-delta spike and mixture components.
  • A. Derivation of Approximate Objective: Combining these approximations yields the tractable objective used for SQS optimization.
  • A. Derivation of Approximate Objective: The mixture KL is empirically approximated using only the dominant maximum-weight component, indexed by k*.A small temperature is used to avoid a flat posterior and large training–inference differences.

B. Proof of Theorem 1

The theorem analyzes sparse and quantized fully connected ReLU networks under specified structural, smoothness, boundedness, and variational-family conditions. Its proof bounds approximation and KL terms to establish high-probability convergence in squared Hellinger distance.

  • B. Proof of Theorem 1: The theoretical model is an L-hidden-layer fully connected ReLU network with layer widths, weights, biases, and parameters explicitly defined.
  • B. Proof of Theorem 1: Sparse and quantized weights contain at most s nonzero entries and K distinct nonzero values, with one-hot indexing for quantization choices.
  • B. Proof of Theorem 1: The proof assumes growing network dimensions, a 1-Lipschitz activation, Hölder smoothness, bounded mixture means, and restrictions on the variational family.
  • B. Proof of Theorem 1: Under the theorem’s conditions and tuning of λ, the variational estimator converges with high probability in squared Hellinger distance.The proof invokes bounds from Lemmas 1 and 2 after selecting λ to satisfy the required assumptions.
  • B. Proof of Theorem 1: Lemma 1 supports approximation of the true function by spike-and-GMM priors, while Lemma 2 states that estimation error vanishes as n→∞.
  • B. Proof of Theorem 1: The proof proceeds by bounding approximation, integral, and KL-divergence terms under the stated sparsity and network conditions.

B.2. Proof of Lemma 2

The proof of Lemma 2 uses variational KL characterization, truncation, and high-probability bounds to control the terms needed for the convergence result.

  • B.2. Proof of Lemma 2: The proof also uses variational-distribution definitions and posterior-sample bounds to complete the lemma.
  • B.2. Proof of Lemma 2: Lemma 4 supplies a Donsker–Varadhan variational characterization of KL divergence used in the proof.
  • B.2. Proof of Lemma 2: The proof truncates the variational distribution and prior to the set H_n(θ) before applying the bounds.
  • B.2. Proof of Lemma 2: A high-probability bound controls the first term, while the remaining terms are bounded using d²(Pθ,P0)≤1.
  • B.2. Proof of Lemma 2: Substituting the component bounds into the main inequality yields the claimed high-probability result.

C. Implementation of SQS

SQS initializes its quantization parameters from K-means clusters and uses model-specific settings for sparse, low-bit compression. Training includes model-dependent fine-tuning, pruning schedules, temperatures, and Gaussian-component choices.

  • Pretrained model setting: For Llama3.2-1B compression on SST-2, omitting fine-tuning before SQS significantly degrades performance.The pretrained model is downloaded from Hugging Face, fine-tuned on SST-2, and then compressed.
  • Initialization: SQS initializes each layer’s Gaussian-mixture parameters by clustering weights into K groups and computing each group’s empirical mean, standard deviation, and mixture proportion.The groups are assumed to be disjoint and cover all weights in the selected layer.
  • Training settings: The temperature τ′ is halved halfway through training to stabilize optimization and sharpen retention probabilities around learned optima.The initial τ′ is set to a fixed constant during training.
  • Hyperparameter configuration: The number of Gaussian components varies by model: K=4 for ResNet, K=16 for BERT-base, and K=64 for Llama3.2-1B and Qwen2.5-0.5B.Bits is defined as log2 K, and larger K reduces performance drop at the cost of lower compression rate.
  • Training settings: Training uses AdamW, polynomial pruning schedules, fixed pruning learning rate 0.012, and model-dependent quantization learning rates.The quantization learning rate is 5 × 10−4 for ResNet-18 and 5 × 10−5 for the other listed models.

D.1. Experiment settings for benchmark with all baselines

Benchmark experiments evaluate SQS on ResNet, BERT-base, and selected language models using compression rate and task-performance metrics. The reported settings combine fine-tuning, pruning, and Gaussian-mixture quantization, while larger LLMs are excluded because of hardware limits.

  • ResNet: ResNet compression uses fine-tuning for 10 epochs, K=4 components per layer, and 50% sparsity, achieving 32× compression across three ResNet models.The experiments use ResNet architectures on CIFAR-10 and CIFAR-100.
  • BERT-base: BERT-base compression is fine-tuned on the training dataset and completed within 3 epochs, with F1-score drop and compression rate as evaluation metrics.This setting parallels the fine-tuning-based evaluation used for the benchmark.
  • BERT-base: 32× compression is achieved for BERT-base by using K=16 Gaussian components and pruning 75% of parameters.Layer-wise quantization is combined with unstructured pruning.
  • Llama and Qwen: Llama3.1-8B and Qwen2.5-7B are not evaluated because the available hardware cannot run these very large models.The limitation concerns the benchmark’s model scale rather than the compression procedure itself.

D.2. Experiment settings for ablation studies for SQS method

The ablation studies compare Gaussian and Spike-and-GMM priors, define the Bits metric, and visualize long-tailed weight distributions in Llama3.2 and Qwen2.5. These experiments examine prior choice and the weight-distribution structure relevant to SQS.

  • Impact of different priors: The prior ablation replaces the proposed Spike-and-GMM prior with a zero-mean Gaussian prior and replaces delta distributions in the variational family with Gaussian distributions.The resulting comparison is summarized in Table 4.
  • Impact of different priors: The ablation section compares the Gaussian-prior alternative with the proposed Spike-and-GMM prior.The comparison is presented as an experiment rather than as a stated theoretical equivalence.
  • Evaluation metrics: Bits is defined as log2 K, where K denotes the number of shared weight vectors.This metric connects the number of Gaussian components to the reported quantization precision.
  • Weight distributions: The visualizations show long-tailed weight distributions across different layers of Llama3.2 and Qwen2.5.Llama3.2 distributions appear in Figures 6 and 7, while Qwen2.5 distributions appear in Figures 4 and 5.
Loading 2510.08999v2…