PaperlayerView exact revision on arXiv

Source-linked AI summary

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

Mohammad Mozaffari

arXiv:2608.24070v1cs.AIcs.DCcs.LGcs.PF

TL;DR

LLM training and deployment incur substantial computational, memory, and environmental costs, while isolated compression techniques face accuracy-efficiency limits. This thesis introduces the Compression Trinity, jointly applying sparsity, quantization, and low-rank approximations across the LLM lifecycle. Across its methods, the framework reports faster training or inference and improved compressed-model accuracy, including up to 2.57× faster training, 1.25× pretraining speedup, and 3.97 percentage-point zero-shot gains.

  • Problem

    LLM training and deployment require substantial computational, memory, and environmental resources, while isolated compression methods inadequately address the multifaceted efficiency bottleneck.

  • Method

    The thesis jointly applies sparsity, quantization, and low-rank approximations to optimizers, model architectures, and post-training compression across pretraining and inference.

  • Results

    Across experiments, the methods achieve up to 2.57× faster training, 1.25× pretraining speedup, and 3.97 percentage-point zero-shot accuracy gains without fine-tuning.

  • Takeaways & Limitations

    The reported results support applying the three compression pillars jointly across LLM lifecycle stages to improve efficiency and compressed-model performance.

  • Takeaways & Limitations

    In zero-shot settings, sparsity alone can leave an accuracy gap from dense models, indicating a ceiling without other Trinity pillars.

Abstract

from arXiv · show

Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.

Introduction

LLM deployment is constrained by high computational, memory, and environmental costs, while isolated compression methods inadequately address these challenges. The thesis proposes jointly applying sparsity, quantization, and low-rank approximations across pretraining and inference to improve efficiency while recovering accuracy.

  • LLM training and inference require substantial computational, memory, and environmental resources, making scalable deployment difficult.
  • Traditional sparsity, quantization, and low-rank approximation methods are often applied independently, limiting their ability to address multifaceted efficiency bottlenecks.
  • The Compression Trinity combines sparsity, quantization, and low-rank approximations, respectively targeting computation, memory bandwidth, and lost accuracy.
  • The thesis applies the Trinity across pretraining and post-training compression through MKOR, SLOPE, OPTIMA, PATCH, and SLIM.
  • MKOR reduces second-order update complexity from O(d^3) to O(d^2), while accelerating convergence over KFAC by up to 1.85x.
  • SLOPE reaches up to 1.25x pretraining and 1.54x inference speedups using double-pruned sparsity and low-rank lazy adapters.
  • OPTIMA improves zero-shot performance by up to 3.97% without fine-tuning, while PATCH delivers up to 1.38x speedup and 2.96% higher accuracy than 2:4 pruning.

2.2 Hardware Constraints and the Roofline Model

GPU execution is constrained by the interaction between memory bandwidth and compute throughput, formalized by the Roofline model. The thesis maps these bottlenecks to sparsity for reducing computation and quantization for reducing memory traffic.

  • GPU execution speed is governed by memory bandwidth and compute throughput, which define the Roofline model’s performance limits.
  • In the memory-bound region, performance is limited by memory bandwidth, so quantization improves efficiency by moving fewer bytes.
  • In the compute-bound region, performance is limited by peak FLOPs, making sparsity useful because it skips calculations involving zero weights.
  • H100 GPUs provide higher peak FLOPs than A100 GPUs, but slower memory-bandwidth scaling shifts the Roofline knee rightward.Algorithms therefore require higher arithmetic intensity to leave the memory-bound region on H100.
  • Sparse Tensor Core Acceleration: The 2:4 format requires at least two zeros in every contiguous four-element reduction block and stores the remaining values with metadata indices.
  • Sparse Tensor Core Acceleration: Applying 2:4 sparsity during training is difficult because transposition misaligns backward-pass weights with the Sparse Tensor Core’s expected read order.This alignment challenge motivates custom pretraining strategies.

2.6 The Compression Trinity as a Solution Framework

The Compression Trinity combines sparsity, quantization, and low-rank approximations because isolated techniques encounter distinct efficiency or accuracy limits. The thesis applies this framework across optimizer design and model compression, with MKOR demonstrating its pretraining benefits.

  • The Compression Trinity jointly applies sparsity, quantization, and low-rank approximations to address complementary LLM-efficiency challenges.Sparsity reduces computation, quantization reduces data movement, and low-rank approximations recover accuracy lost through compression.
  • Low-rank approximations represent matrices as products of smaller matrices and bridge hardware efficiency with sample efficiency.
  • Applying compression pillars in isolation is limited because aggressive sparsity and quantization can cause accuracy degradation or performance collapse.
  • The thesis uses OPTIMA and PATCH as staged sparsity-focused steps before combining sparsity with quantization and low-rank approximation.
  • MKOR: MKOR applies the Trinity inside the optimizer using sparse and low-rank curvature approximations with quantized internal computations.
  • MKOR: MKOR reduces factor-inversion complexity from O(d^3) to O(d^2), reduces communication complexity to O(d), and updates second-order information up to 100 times more frequently.
  • MKOR: MKOR reduces BERT-Large-Uncased training time from 8 hours to 3 hours on 64 A100 GPUs, a speedup of up to 2.57×.
  • MKOR: MKOR uses O(d^2/2) memory for inverse factors while preserving momentum, compared with KFAC’s O(4d^2) memory overhead.

3.4 Experimental Results

Experiments evaluate MKOR and related optimizers across BERT, ResNet-50, SQuAD, and GLUE settings, measuring accuracy, convergence, timing, and overhead. MKOR achieves faster convergence and lower overhead, with benefits varying by model dimension and covariance structure.

  • Experimental setting: The BERT experiments use Wikipedia and Toronto BookCorpus pretraining data, with incomplete availability of the latter producing slightly lower baseline accuracies than the original BERT results.Reported timings and speedups cover pretraining; fine-tuning time is omitted as negligible relative to pretraining costs.
  • Question answering and convergence: MKOR-H reaches the target accuracy in 600 steps, reducing LAMB’s step count by 2.6× and achieving 2.57× speedup over LAMB.It also achieves 1.75× speedup over KAISA.
  • Classification: MKOR and MKOR-H match the baseline average GLUE metric in 600 steps, reducing the required steps by 2.6×; both achieve 2.57× end-to-end speedup.MKOR with 1500 steps reaches a new state-of-the-art GLUE accuracy on BERT-Large Uncased.
  • ResNet-50: MKOR reaches 75.9% on ResNet-50 in 57 epochs versus 88 for SGD, achieving 1.49× speedup and 1.04× faster convergence than KAISA.HyLo is not compared because it does not reach the target accuracy, attaining 75.6% with tuning.
  • Complexity and timing: MKOR’s overhead is up to 1.5× lower than KFAC/KAISA, while its factor inversion cost is substantially reduced for large models.The experiments attribute this to MKOR’s lower computational complexity, particularly for BERT-Large Uncased.
  • Covariance structure: Rank-1 covariance approximations show low error in the experiments, while higher-rank approximations are reported as unnecessary in practice.The observed covariance matrices remain low-rank under the experimental settings.

3.5 Conclusion

The chapter presents MKOR and SLOPE as complementary applications of the Compression Trinity to pretraining: optimizer-level convergence acceleration and per-iteration sparse training efficiency. MKOR reduces second-order overheads, while SLOPE combines double-pruned N:M sparsity with lazy low-rank adapters to preserve quality and deployment efficiency.

  • MKOR: MKOR applies sparsity, low-rank updates, and stability mechanisms to second-order optimization, reducing inversion complexity from O(d^3) to O(d^2).It also reduces communication overhead to O(d).
  • MKOR: Up to 2.57× faster training was achieved for large language models, while MKOR outperformed state-of-the-art first- and second-order optimizers.
  • SLOPE: SLOPE addresses dense linear-layer costs with a double-pruned backward pass that enables N:M sparsity in both forward and backward computations.The method transposes an already sparsified weight matrix before imposing a second N:M pattern, avoiding transposable-mask search.
  • SLOPE: Lazy low-rank adapters are introduced only during the final 1% of pretraining iterations to recover accuracy while keeping the base model sparse.The sparse weights and dense adapters remain parallel computational paths rather than a merged tensor.
  • SLOPE: SLOPE reports 1.25× end-to-end training speedup and 1.54× inference speedup on LLMs with billions of parameters.The speedups come from jointly optimized sparse kernels and low-rank calls.

4.3 Sparse Plus Low-rank Pretraining of LLMs

SLOPE combines structured N:M sparsity across forward and backward computations with low-rank adapters to accelerate LLM pretraining while recovering accuracy. The method reduces memory use and improves training and inference efficiency, though evaluation is primarily conducted on smaller models for accuracy.

  • Double-pruned backward pass: SLOPE double-prunes weight tensors along row and column dimensions to accelerate both forward and backward computations.The formulation uses N:M sparsity in both dimensions and is designed to productively accelerate FWD and BWD-2 operations.
  • Memory footprint: N:M sparsity reduces computational cost and memory footprint, but requires auxiliary metadata recording nonzero locations.The storage trade-off is analyzed through the extra bits needed for sparse indices.
  • Double-pruned backward pass: 12.5%, 9.375%, and 3.39% additional sparsity arise from double pruning under 1:2, 2:4, and 2:8 patterns, respectively.The surplus of zero elements decreases as M increases, reducing computational errors.
  • Runtime and scaling results: SLOPE reports 12% faster inference, 4% faster training, and up to 6% higher end-to-end inference speed from runtime optimizations.Reported speedup and memory-reduction results extend from 2.6B to 66B models, while accuracy evaluation focuses on models up to 774M parameters.
  • Quality and convergence: SLOPE maintains competitive quality across sparsity configurations, outperforming Wanda in perplexity and Extended SR-STE in accuracy.Accuracy declines as sparsity increases, and the first model blocks are more sensitive than later blocks to pruning during pretraining.
  • Lazy low-rank adapters: Low-rank adapters added during only the final 1% of training recover pretraining accuracy while limiting compute and memory overhead.Higher adapter ranks improve performance but require more computation.

4.5 Conclusion

The chapter presents SLOPE and OPTIMA as complementary Compression Trinity methods for accelerating pretraining and enabling accurate post-training compression under constrained resources.

  • SLOPE: SLOPE combines double-pruned backward passes with lazy low-rank adapters to accelerate N:M sparse training while recovering model capacity.The method targets both forward and backward passes and applies adapters during the final 1% of training.
  • SLOPE: 1.25× pretraining speedup and 1.54× inference speedup are achieved by SLOPE’s joint sparsity and low-rank approach.
  • OPTIMA: OPTIMA addresses one-shot post-training pruning by solving column-wise constrained quadratic programs after mask selection.Its shared-Hessian formulation enables parallel accelerator execution and per-column global optimality.

5.2 Additional Related Work

Prior pruning methods trade reconstruction optimality against computational feasibility, motivating OPTIMA’s globally optimal, column-wise quadratic-program formulation.

  • Post-training pruning: One-shot pruning uses small calibration datasets, but existing methods often choose between efficiency and reconstruction optimality.
  • OPTIMA: OPTIMA reformulates post-pruning weight reconstruction as independent, accelerator-friendly quadratic programs for each weight-matrix column.The formulation uses the shared Hessian H = X^T X across columns.
  • OPTIMA: Shared-Hessian structure provides per-column global optimality while allowing the problems to be parallelized on GPUs and TPUs.

5.5 Experiments

Experiments across model families, sparsity regimes, and downstream tasks show that OPTIMA consistently improves post-pruning accuracy, with larger gains at higher sparsity.

  • Evaluation: OPTIMA is evaluated on LLaMA, Gemma, and Qwen model families using zero-shot downstream tasks, perplexity, and structured sparsity settings.
  • Unstructured sparsity: 1.80% average accuracy improvement is achieved across six downstream tasks for Gemma-3-1B at 50% unstructured sparsity.
  • 2:4 sparsity: At 2:4 sparsity, pruning only MLP weights yields 38% to 41% end-to-end sparsity while OPTIMA consistently improves accuracy.
  • 2:4 sparsity: 2.53% average accuracy gains are reported across downstream tasks for LLaMA-3.2-1B under the evaluated pruning setup.
  • High sparsity: 4.19% absolute gain is obtained on Qwen-2.5 3B when OPTIMA improves Wanda from 43.67% to 47.86% at 60% sparsity.
  • Overall findings: OPTIMA’s gains persist without fine-tuning and reach up to 3.97 percentage points across LLM families, including at sparsity levels of at least 60%.

6.2 Introduction

PATCH addresses the rigidity of fixed 2:4 sparsity by learning hardware-friendly hybrid masks with effective global sparsity anywhere from 0% to 50%. It targets the trade-off between flexible unstructured pruning and accelerable structured sparsity.

  • PATCH motivation and design: PATCH bridges these extremes by learning the sparsity pattern itself during training rather than optimizing values under a fixed mask.The approach uses a relaxed training budget to move beyond the zero-training regime.
  • PATCH motivation and design: Unstructured sparsity offers flexible allocation but irregular memory access patterns hinder practical GPU acceleration.Semi-structured sparsity accelerates hardware but imposes rigid layout expectations.
  • PATCH motivation and design: Fixed 2:4 sparsity enforces a uniform 50% ratio across layers, failing to account for differing component sensitivity and often causing accuracy loss.The fixed allocation contrasts with evidence that optimal sparsity should be non-uniform.
  • PATCH motivation and design: PATCH learns tile-level hybrid masks that designate each tile as dense or 2:4 sparse, enabling global sparsity ratios from 0% to 50%.Dense tiles preserve accuracy in critical regions, while 2:4 tiles retain hardware-friendly sparsity elsewhere.
  • PATCH optimization variants: PATCH supports joint optimization of tile locations and within-tile 2:4 patterns, or a lower-budget variant that learns only dense-tile locations with fixed sparse masks.Both strategies retain the dense/2:4 hybrid structure.
  • Related pruning approaches: PATCH is evaluated as a hardware-oriented extension of existing pruning approaches, including local, global, one-shot, and learned structured methods.The comparison reflects the broader trade-off between pruning quality and computational efficiency.

6.6 Efficient deployment of PATCH

PATCH combines learnable hybrid sparsity with tile-level execution support to provide flexible accuracy–efficiency trade-offs on modern GPUs. Across models and tasks, it improves quality over fixed and one-shot pruning while delivering practical speedups, but requires a fine-tuning budget.

  • Efficient deployment: STOICC extends Triton with hybrid tile-level sparsity, providing a backend suitable for accelerating PATCH.Existing tools primarily support dense computation or fixed 2:4 sparsity, whereas STOICC handles hybrid sparse–dense tiles.
  • Experimental results: PATCH progressively aligns with dense-model performance as sparsity decreases from 45% to 25%, producing accuracy gains and perplexity reductions.The flexible dense-tile allocation drives the reported quality–sparsity trade-off.
  • Experimental results: Across eight downstream tasks and WikiText2 perplexity, PATCH narrows the gap between hybrid and unstructured sparsity without a significant difference.The results indicate that hybrid sparsity can approach the quality of unstructured sparsity while remaining more hardware-compatible.
  • Efficiency trade-offs: At 25%, 35%, and 45% sparsity, GPU memory falls to 0.76×, 0.68×, and 0.59× of the fully dense model, respectively.The corresponding speedups are 1.18×, 1.27×, and 1.38× versus the dense baseline.
  • Limitations and scope: PATCH requires a fine-tuning budget and introduces computational overhead, making it unsuitable for instant on-device adaptation.OPTIMA instead targets the zero-training regime, while PATCH targets settings where resources permit training.
  • Transition to SLIM: SLIM extends PATCH’s sparsity work by combining optimized sparsity with quantization and low-rank approximation in a unified one-shot framework.This integration addresses compounded errors from applying aggressive sparsity and quantization together.

7.2 Related work

Related work spans pruning, quantization, low-rank adaptation, and sparse-plus-low-rank decomposition. SLIM combines these ideas sequentially, adding quantization and saliency-weighted low-rank correction while avoiding costly retraining.

  • Compression methods: SLIM combines model pruning and quantization with zero-shot low-rank adapters intended to recover lost accuracy.The framework targets compressed models without requiring the full fine-tuning regimes used by several prior adapter methods.
  • Quantization: Quantization methods reduce precision using techniques such as RTN, OPTQ, AWQ, SmoothQuant, OmniQuant, AffineQuant, and QuaRot, but low-bit accuracy remains challenging.Outliers and low-bit quantization errors motivate specialized calibration, scaling, clipping, or rotation strategies.
  • Low-rank adaptation: Prior low-rank methods mitigate quantization or sparsity errors, but some perform poorly with combined sparsity or require hundreds of millions of fine-tuning tokens.Sparse-plus-low-rank adapters can also add extra sparse matrix multiplication during inference.
  • Sparse-plus-low-rank decomposition: Sparse-plus-low-rank decomposition has improved accuracy–compression trade-offs over pure pruning or pure low-rank factorization in earlier model-compression work.Related approaches include RPCA, OATS, HASSLE-free, and 3BASiL formulations.
  • SLIM formulation: SLIM formulates weights as W ≈ W_C + LR, where W_C is sparse and quantized and LR is a low-rank correction.Its sequential pipeline quantizes, sparsifies, and then computes the low-rank correction.
  • SLIM formulation: SLIM’s saliency-weighted SVD incorporates input statistics into the decomposition, unlike standard RPCA but similarly to outlier-aware scaling in OATS.The method derives the low-rank component analytically rather than solving the sparse and low-rank terms jointly by convex optimization.
  • SLIM contribution: SLIM-LoRA derives low-rank adapters from compression error through saliency-weighted SVD, enabling a one-shot solution without iteration.The approach is presented as a practical, efficient method tailored to compression errors in LLMs.

7.5 Experimental results

SLIM evaluates joint sparsity, quantization, and low-rank adaptation across OPT and LLaMA-2 models, showing improved accuracy, efficiency, and controllable compression tradeoffs.

  • Evaluation setup: SLIM is evaluated on OPT and LLaMA-2 model families using zero-shot downstream tasks and language-modeling benchmarks.Evaluation includes MMLU, Piqa, Arc-Easy, Arc-Challenge, WinoGrande, and OpenBookQA, alongside WikiText2.
  • Low-rank adaptation: SLIM-LoRA improves over Naive-LoRA by incorporating weight saliency, while SLIM-LoRAQ reduces adapter overhead with minimal accuracy impact.Lightweight fine-tuning further improves both methods, with larger gains for the saliency-aware SLIM-LoRA.
  • Integrated compression: Combining PATCH with quantization and low-rank approximation enables controllable tradeoffs between compression ratio and model quality.Applying OPTIMA with SLIM-LoRA further boosts accuracy on WikiText2.
  • Accuracy results: SLIM and its variants consistently outperform dense models and other compression techniques at the same parameter size.The results indicate that compressed models can achieve higher accuracy than dense models of equivalent size.
  • Quantization: Adding low-rank adapters to Group AbsMax significantly boosts accuracy and can match or exceed Group AbsMax with low-rank adapters in integrated SLIM variants.SLIM-Quant alone is not designed for high accuracy, whereas its integration with SLIM variants produces comparable or better results.
  • Efficiency: SLIM provides state-of-the-art accuracy after lightweight fine-tuning and can process 13B parameter models in 14 hours on a single GPU.The speedup results also show greater gains for larger self-attention and feed-forward matrices.
  • Quantization: 4-bit weight quantization combined with sparsity achieves better compression results than using 2-bit quantization alone.The experiments use a 4-bit weight-only scheme with group size 128.

7.6 Conclusion

The thesis presents the Compression Trinity as a life-cycle framework combining sparsity, quantization, and low-rank approximation across training and deployment. Its methods address compounded compression error, while remaining subject to theoretical, engineering, fairness, and hardware constraints.

  • SLIM: SLIM uses mathematically derived low-rank error correction to recover information lost through aggressive sparsity and quantization.This addresses the compounded-error challenge in joint compression.
  • Core thesis: The Compression Trinity jointly attacks compute FLOPs, memory bandwidth, and parameter redundancy through sparsity, quantization, and low-rank approximation.The framework is presented as applicable across the model life-cycle rather than only to post-training optimization.
  • Training: MKOR accelerates convergence by up to 2.57× over first-order baselines and 1.75× over KFAC.It applies sparse and low-rank structure to second-order optimization.
  • Training and inference: PATCH makes sparsity ratios continuous and adaptive from 0–50%, preserving density in information-critical components.SLIM provides the unified inference setting that combines the three pillars.
  • Results: Fully compressed models improve accuracy by up to 5.66% over state-of-the-art methods and can outperform dense models at equivalent parameter budgets.Specific configurations achieve up to 0.6% higher accuracy than uncompressed dense models at equal parameter budgets.
  • Limitations: The thesis’s implementations face engineering and portability barriers because Trinity pipelines are more complex and rely on NVIDIA-specific sparsity and custom kernels.Transferability to TPUs, AMD accelerators, and CPUs remains unproven.
  • Limitations: Aggregate benchmark accuracy may hide degradation for low-resource languages or minority demographic groups.The thesis calls for fairness audits across languages, dialects, demographic groups, and downstream tasks before high-stakes deployment.

Appendix A

Appendix A details the derivations, numerical behavior, scalability, and learning-rate robustness of MKOR and related second-order optimizers. It also reports optimizer comparisons and approximation behavior during training.

  • Numerical stability: MKOR avoids numerical instabilities during matrix inversion, and its computational complexity is independent of the condition number.KFAC factors can become singular with large condition numbers, motivating MKOR’s alternative inversion behavior.
  • Learning-rate robustness: MKOR converges across a wider range of learning rates than first- and second-order comparison methods.Other methods may diverge, ripple, or lose performance as learning rates vary.
  • Scalability: MKOR scales strongly on BERT-Large-Uncased across up to 64 GPUs.The appendix presents scalability results for distributed training.
  • Low-rank structure: MKOR’s factor eigenvalues decay as training converges, making rank-1 approximations more effective.The appendix attributes this decay to increasing dependence among neurons and resulting linear dependence in activations and input gradients.
  • Optimizer comparisons: 1.58× is MKOR’s reported speedup over KAISA, alongside 1.26× over HyLo-KIS and 1.31× over SGD.These optimizer comparisons are reported as relative speedups.

B.2 cuSPARSELt Initialization Overhead: Static vs. Dynamic Sparsity

This appendix examines sparse-training overheads and pruning choices. Static masks avoid repeated setup costs, while pruning weights and adding low-rank adapters improve the reported performance profile.

  • Initialization overhead: Setup overhead is significantly larger than matrix multiplication time in the cuSPARSELt SpMM pipeline.With static sparsity masks, SLOPE pays the setup cost once, making it negligible across repeated training and inference multiplications.
  • Static vs. dynamic sparsity: Dynamic sparsity can reduce speedups or cause slowdowns because changing masks incurs substantial setup overhead.The appendix contrasts this behavior with static masks, whose setup cost is amortized.
  • Training behavior: Higher sparsity ratios generally produce higher training loss in both BERT-Large-Uncased pretraining phases.The reported loss/perplexity gap does not directly correlate with downstream accuracy drops.
  • Pruning choices: Static pruning achieves lower perplexity than dynamic pruning, while pruning weights performs better than pruning inputs.Pruning output gradients causes divergence within a few iterations and is omitted from the figure.
  • Accuracy recovery: Adding low-rank adapters improves the accuracy of all evaluated sparse pretraining methods.SLOPE outperforms GPT2-Half on average, while dynamic SR-STE performs worse than GPT2-Half in the reported comparison.

B.13 Comparison of Depth and Width Pruning

This appendix compares depth and width pruning and supplies supplementary analyses for OPTIMA and PATCH. It reports preliminary pruning comparisons, calibration-sample behavior, and hardware-oriented mixed-sparsity results.

  • Depth and width pruning: Width pruning receives an architectural advantage because the Self-Attention module’s parameter count remains intact.The Up-Sample and Down-Sample aspect ratio was reduced to half in the tested configurations.
  • Depth and width pruning: Depth- and width-pruning show no significant difference in preliminary pretraining loss curves.The comparison covers LLaMA-2-7B, Gemma-2-2B, and Gemma-2-9B configurations.
  • OPTIMA: OPTIMA shows consistent, though not significant, model-quality improvement as calibration samples increase.Wanda and SparseGPT exhibit stochastic behavior with increasing sample counts in the reported comparison.
  • PATCH: 25%–45% sparsity yields 1.18×–1.38× end-to-end acceleration over the dense baseline.The measurements use the heterogeneous mixed-sparsity approach for LLaMA-2 7B inference.
  • PATCH: PATCH variants maintain robust performance across diverse zero-shot tasks while preserving hardware-friendly sparsity patterns.The supplementary results cover larger and smaller language models at various sparsity levels and pruning methods.

E.1 Notations

The appendix evaluates SLIM’s combined pruning, quantization, and low-rank-adapter design across accuracy, perplexity, efficiency, and robustness settings.

  • Robustness: SLIM shows minimal accuracy loss from 8-bit input quantization and low sensitivity to calibration-sample count and calibration-dataset choice.Weights use FP8 quantization, while inputs use AbsMax uniform quantization with one parameter per input tensor.
  • Fine-tuning: Lightweight fine-tuning improves all baselines, with SLIM-LoRA producing the largest improvements through its saliency-based design.The experiments use low-rank adapters while keeping quantized weights frozen, reducing memory and computational costs.
  • Language modeling: SLIM achieves superior performance across 4-bit quantized, 2:4 sparse, unstructured sparse, pruning-only, and quantization-only WikiText2 settings.The appendix reports perplexity for each setting, with lower perplexity indicating better performance.
  • Compression comparisons: SLIM combines pruning and quantization more effectively than quantization-only settings at comparable approximately 8× model-size reduction.The comparison covers 2-bit weights without sparsity versus 4-bit weights with 50% unstructured or 2:4 sparsity.
  • Efficiency: Larger models achieve higher SLIM speedups on NVIDIA A100-40GB GPUs, while Sparse Marlin can slow smaller LLaMA-2-7B models.The reported slowdown is attributed to Sparse Marlin’s sparse quantized matrix-multiplication kernels and is not specific to SLIM.
  • Sparsity: SLIM-LoRA with SLIM-Quant maintains competitive accuracy up to 60% sparsity, while other methods degrade at lower sparsity levels.Higher sparsity ratios also increase perplexity, exposing a compression–accuracy trade-off.
Loading 2608.24070v1…