Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

1.4 Thesis Contributions and Roadmap

• Because each technique independently reaches an accuracy or efficiency wall, the thesis frames compression as a multidimensional balancing problem requiring joint application of the three methods.The introduction identifies the high computational, memory, and environmental costs of training and deploying modern LLMs as the motivating challenge.Source: Introduction · p. 18, §1.3 · p. 21, §1.4 · p. 21

Introduction

• The Compression Trinity jointly applies sparsity, quantization, and low-rank approximations because their complementary roles address computation, memory bandwidth, and model expressivity.Sparsity reduces computational load, quantization minimizes memory bandwidth, and low-rank projections recover accuracy and capacity without restoring eliminated hardware overheads.Source: Introduction · p. 18

1.1 LLM Life-cycle Stages

• LLM development emphasizes pretraining and inference as primary computational bottlenecks, enabling training acceleration or faster optimizer convergence and post-training compression for efficient deployment.Inference often requires low latency on resource-constrained hardware, and both stages share dense matrix-matrix multiplication as a computational bottleneck.Source: §1.1 · pp. 18–19

1.2 Compression Techniques

• The three compression pillars target distinct tradeoffs: sparsity removes weights and FLOPs, quantization lowers numerical precision and memory, and low-rank factorization exploits matrix redundancy.Low-rank factorization represents W ∈ R^m×k as L ∈ R^m×r and R ∈ R^r×k with r ≪ m, k.Source: §1.2 · p. 19, §1.2.1 · p. 19, §1.2.2 · p. 20

1.2.1 Sparsity

• N:M sparsity preserves accuracy while enabling hardware acceleration, but sparsity remains challenging because optimal patterns and compensating weight updates are difficult to determine.Unstructured sparsity is flexible but hardware-unfriendly, whereas structured sparsity is regular but can substantially damage accuracy.Source: §1.2.1 · p. 19

1.2.2 Quantization

• 4-bit quantization yields an 8x memory reduction over 32-bit weights and can use faster integer compute, but low-bit representations and outliers can cause substantial accuracy degradation.Mapping high-precision weights to low-bit representations while minimizing error is non-trivial, and quantization alone eventually limits total compression.Source: §1.2.2 · p. 20

1.2.3 Low-rank Approximations

• Low-rank approximations reduce memory and compute by exploiting overparameterized matrices, but aggressive compression fails on inherently high-rank information and cannot replace sparsity or high-precision quantization.LoRA popularizes decomposing large matrices into two thin factors with rank r much smaller than both dimensions.Source: §1.2.3 · p. 20

1.3 The Failure of Isolated Compression

• At 8x compression, single-pillar 2-bit quantization or 87.5% sparsity reduces LLaMA-2-7B reasoning accuracy to approximately 31%, whereas 4-bit quantization plus 50% sparsity recovers 52.38%.The comparison is reported for average zero-shot accuracy across eight tasks; the hybrid achieves the same compression ratio.Source: Table 1.2 · p. 21, §1.3 · p. 21

1.4 Thesis Contributions and Roadmap

• MKOR reduces second-order update complexity from O(d^3) to O(d^2) and communication from O(d^2) to O(d), outperforming KFAC by up to 1.85x on BERT-Large pretraining.It combines block-diagonal sparsity, rank-1 updates, and quantized 16-bit inverse factors while maintaining numerical stability.Source: §1.4 · pp. 21–22
• SLOPE accelerates OPT-66B end-to-end training by up to 1.14x and inference by 1.34x while reducing training memory by 0.77x.Its double-pruned backward pass enables N:M sparsity, while low-rank adapters are added only during the final 1% of pretraining iterations.Source: §1.4 · p. 22
• OPTIMA improves zero-shot performance by up to 3.97% absolute accuracy without fine-tuning under a zero-training constraint.It replaces pruning updates in methods such as Wanda or SparseGPT with globally optimal row-wise quadratic-program updates based on a shared layer Hessian.Source: §1.4 · p. 22
• PATCH delivers 1.18x–1.38x end-to-end speedup and up to 2.96% higher accuracy than state-of-the-art 2:4 pruning on LLaMA-2 7B.Its learnable tile-level mask assigns dense or 2:4-sparse tiles, producing an effective sparsity ratio between 0% and 50% and remaining composable with other Trinity pillars.Source: §1.4 · p. 22
• Up to 5.66% accuracy improvement and 4.3x layer-wise speedup result from SLIM’s one-shot combination of 4-bit quantization, 2:4 sparsity, and low-rank approximation on LLaMA-2-7B.SLIM computes low-rank adapter values using a saliency function to compensate for compounded quantization and sparsity error.Source: §1.4 · pp. 22–23

2.1 The Transformer Bottleneck: Linear Layers

• Transformer blocks comprise self-attention and feed-forward networks, but dense GEMMs in their linear layers contain most computations and parameters.Attention uses Q, K, V, and O projections, while SwiGLU FFNs use up-, gate-, and down-projections, making linear layers the primary optimization target.Source: §2.1 · pp. 24–25
• Each linear layer requires one forward GEMM and two backward GEMMs for weight and input gradients, with the latter multiplying by W rather than W^T.The forward computes Y = XW^T, while backpropagation computes ∇W L = (∇Y L)^T X and ∇X L = (∇Y L)W.Source: §2.1 · p. 25
• The transpose relationship means a sparsity pattern hardware-friendly along W’s rows for the forward pass may be unfriendly along its columns for the backward pass.This asymmetry is identified as a central obstacle for structured compression.Source: §2.1 · p. 25
• 51.8% of LLaMA-3.1-8B training time is consumed by the linear component, making dense linear layers the central efficiency target.Figure 2.2 profiles training and inference on a single NVIDIA H100 GPU; attention accounts for another 9.6%.Source: §2.1 · p. 25, Figure 2.2 · p. 26

2.2 Hardware Constraints and the Roofline Model

• GPU execution is constrained by memory bandwidth and compute throughput, with data movement through HBM, cache, shared memory, and registers shaping efficiency.Algorithms with high data locality reduce costly HBM transfers, while Tensor Cores accelerate matrix-multiply-and-accumulate operations on specialized hardware.Source: §2.2.1 · p. 26

2.2.2 The Roofline Model

• The Roofline model separates memory-bound workloads limited by bandwidth from compute-bound workloads limited by Tensor Core FLOPs, with arithmetic intensity determining the regime.Quantization reduces bytes moved in the memory-bound region, whereas reducing operations improves compute-bound performance; the H100’s higher compute peak shifts the knee rightward.Source: §2.2.2 · p. 27

2.2.3 Sparse Tensor Core Acceleration

• 2:4 structured sparsity compresses matrices by 50% and theoretically doubles Sparse Tensor Core peak throughput by skipping multiplications involving zeros.The format requires at least two zeros in every contiguous block of four elements, but backward-pass transposition misaligns the pattern and motivates custom training strategies.Source: §2.2.3 · pp. 27–28

2.3 The Solution Space: Two Frontiers of Acceleration

• Acceleration decomposes into sample efficiency, reducing iterations during training, and hardware efficiency, reducing time per iteration in training and inference.Hardware efficiency targets compute-bound FLOPs or memory-bound traffic according to the Roofline model.Source: §2.3 · p. 29

2.4 Strategy 1: Algorithmic Efficiency

• Algorithmic efficiency seeks fewer updates by improving the optimization trajectory from initialization to convergence, rather than only accelerating each update.This frontier motivates advanced optimizers as distinct from hardware-level reductions in per-iteration cost.Source: §2.4 · p. 29

2.4.1 First-Order Methods

• First-order methods such as Adam [70] and AdamW require O(d) memory and compute but ignore curvature, causing oscillation in narrow valleys and difficult step-size selection.Their local hyperplane approximation is computationally efficient but can waste iterations in ill-conditioned, non-convex landscapes.Source: §2.4.1 · p. 29

2.4.2 Second-Order Methods

• Second-order methods use the Hessian to form a quadratic local model and rescale gradients, taking larger steps in flat directions and smaller steps in steep directions.Affine invariance theoretically enables convergence in significantly fewer iterations than first-order methods.Source: §2.4.2 · pp. 29–30

2.4.3 The Computational Barrier and Structured Approximations

• Exact second-order optimization is infeasible for LLMs because a d×d Hessian requires enormous storage and O(d^3) inversion, motivating structured approximations such as KFAC [93].Second-order methods trade high sample efficiency against catastrophic computational cost; KFAC approximates the Fisher Information Matrix layer-wise rather than as one dense matrix.Source: §2.4.3 · p. 30
• 2.4.3 shows that KFAC reduces curvature inversion from O(d^3) to O(width^3), but its curvature factors can triple memory relative to Adam, motivating Trinity-based optimizer design.The structured approximation uses H ≈ A ⊗ G and (A⊗G)^−1 = A^−1⊗G^−1, while retaining prohibitive memory costs for modern LLMs.Source: §2.4.3 · p. 30

2.5 Strategy 2: Hardware Efficiency and LLM Regimes

• Advanced optimizers improve sample efficiency but not fixed-step inference, so hardware efficiency requires regime-specific strategies under the Roofline model.The thesis distinguishes compute-bound training and prefill from memory-bound decoding and focuses its structured optimizer discussion on the KFAC family, while noting applicability to alternatives such as Shampoo [50] and Muon [68].Source: §2.5 · p. 30

2.5.1 The Compute-Bound Regimes: Training and Prefill

• Training and prefill are compute-bound because massive parallelism and high arithmetic intensity saturate compute units, making sparsity the premier accelerator by reducing FLOPs.Weights are loaded once and reused across thousands of tokens, so memory bandwidth is not the limiting factor; skipping zero-valued weights lowers training-step and prompt latency.Source: §2.5.1 · p. 31

2.5.2 The Memory-Bound Regime: Inference Decoding

• Decoding is memory-bound because sequential generation repeatedly loads the model with minimal data reuse, making quantization dominant; reducing weights from 16-bit to 4-bit cuts data volume by 75%.Reducing FLOPs offers diminishing returns in this regime, whereas lower bit-width feeds compute units faster.Source: §2.5.2 · p. 31
• The Compression Trinity jointly targets compute and bandwidth bottlenecks, using sparsity to cut FLOPs, quantization to cut data movement, and low-rank approximations to recover accuracy.Sparsity alone leaves memory-bound decoding largely untouched, while quantization alone cannot accelerate compute-bound training and prefill.Source: §2.5.2 · p. 31

2.6 The Compression Trinity as a Solution Framework

• The Compression Trinity jointly uses sparsity to reduce computation, quantization to reduce memory traffic, and low-rank approximations to recover accuracy and support efficient optimization.Sparsity targets FLOP reduction, quantization reduces bit-width and bandwidth, and low-rank structure reduces parameters while mitigating degradation from the other pillars.Source: §2.6 · p. 32

2.7 The Case for a Joint Approach

• The thesis argues that isolated compression methods hit fundamental limits, so it stages robust sparsity advances before unifying the pillars for pretraining and inference.OPTIMA and PATCH strengthen sparsity as a building block, while MKOR and SLOPE target pretraining and SLIM completes the joint inference framework.Source: §2.7 · p. 33

Using Rank-1 Updates

• The chapter’s MKOR work was published at NeurIPS 2023 and developed collaboratively, with Mohammad Mozaffari conceiving the algorithm and leading its implementation and experiments [99].Sikan Li assisted with experiments, while Zhao Zhang and Maryam Mehri Dehnavi supervised the project and contributed to writing and revision.Source: Using Rank-1 Updates · p. 34

3.1 Introduction

• LLM pretraining is limited by convergence cost, intractable exact Hessian computation, and optimizer-state memory, motivating compressed curvature approximations.KFAC uses block-diagonal Fisher approximations, but block inversion scales as O(d^3), forcing infrequent curvature updates.Source: §3.1 · pp. 34–35
• MKOR combines block-diagonal sparsity with rank-1 Sherman–Morrison inverse updates, reducing inversion complexity from O(d3) to O(d2) while synchronizing O(d) vectors instead of O(d2) factors.It preserves momentum history through exponentially averaged curvature updates, unlike Eva, whose memory savings discard accumulated inverse-factor momentum and can impair convergence.Source: §3.1 · p. 35
• MKOR uses half-precision storage and computation for curvature factors, replacing numerically unstable decompositions and inversions with matrix-vector updates that tolerate quantization noise.This integrates the quantization pillar without sacrificing the rank-1 update formulation’s numerical robustness.Source: §3.1 · pp. 35–36

3.3 Methodology

• MKOR-H switches between second-order and first-order optimization according to the loss-reduction rate, combining rapid early convergence with lower later-stage computational overhead.The hybrid variant is designed to maintain efficient resource use throughout pretraining.Source: §3.1 · p. 36

3.1 Introduction

• Up to 2.57× faster, MKOR reduces BERT-Large-Uncased training from 8 hours to 3 hours on 64 A100 GPUs while achieving state-of-the-art GLUE metrics where KFAC fails.MKOR outperforms distributed second- and first-order methods and validates applying the Compression Trinity to optimizer computations.Source: §3.1 · p. 36

3.3.1 The MKOR Algorithm

• The MKOR algorithm updates second-order information, preconditions gradients, and applies a backend optimizer update using rank-1 covariance approximations, norm stabilization, Sherman–Morrison inversion, and gradient rescaling.The norm-based stabilizer addresses factor overflows, underflows, oscillations, and divergence, while rescaling preserves the original gradient norm for effective learning-rate scheduling.Source: §3.3.1 · pp. 37–38
• MKOR reduces factor-inversion computation to O(d2), halves synchronized rank-1 approximation data through quantization, and lowers communication complexity from quadratic to linear.Table 3.1 contrasts these costs with KFAC’s O(d3) inversion and SNGD’s O(b3) inversion, whose costs become comparable to layer dimensions in Transformers.Source: §3.3.1 · pp. 38–39, Table 3.1 · p. 39
• MKOR retains half-precision inverse factors, using O(d^2/2) memory—below KFAC’s O(4d^2)—while enabling momentum accumulation and more stable convergence than Eva.Eva stores O(d) memory but discards factor inverses, whereas MKOR retains them to accumulate momentum across iterations.Source: §3.3.1 · p. 39

3.3.2 Hybrid MKOR

• MKOR-H switches from second-order to first-order optimization based on loss-decrease rates, reducing costly second-order updates as convergence makes their advantage less noticeable.The hybrid design exploits stronger early benefits from second-order methods and their diminishing advantage near convergence.Source: §3.3.2 · p. 39

3.3.3 MKOR Convergence and Stability

• 150× higher inversion-iteration cost and over 98% inversion overhead motivate avoiding stale factors, which can otherwise produce local minima and harm generalization.Increasing inversion frequency can improve convergence, while stale-factor assumptions may fail when the loss landscape changes substantially between iterations.Source: §3.3.3 · p. 40
• MKOR avoids damping and higher-precision fixes: its guaranteed non-zero scalar denominator prevents numerical overflow or underflow, while half-precision operations reduce GPU memory and communication costs by 2×.The method also bounds quantization-error effects and modifies factors toward SGD when exploding gradients are detected, though excessive modification converts MKOR to SGD.Source: §3.3.3 · p. 40

Experimental Limitations

• HyLo cannot execute on a 40GB A100 for a single sample, lacks gradient accumulation at LLM batch sizes up to 64k, and diverges in the reported training process.Eva’s ResNet comparison is also limited because its hyperparameters are unavailable and tuning did not reach the desired accuracy.Source: §3.4 · pp. 42, 45

Large Language Models

• 2.57× speedup over LAMB and 1.75× over KAISA make MKOR-H the fastest SQuAD optimizer while matching baseline accuracy in 600 steps.MKOR-H reduces LAMB’s 1,563 steps by 2.6×; vanilla MKOR and KAISA converge in 1,000 steps, while Eva is 1.69× slower than MKOR-H.Source: §3.4 · p. 42, Table 3.3 · p. 43
• MKOR and MKOR-H achieve the baseline average GLUE metric with 600 steps, reducing steps 2.6×, while both deliver 2.57× end-to-end speedup; MKOR reaches state-of-the-art accuracy at 1,500 steps.After 1,563 steps, KAISA does not converge to baseline average accuracy and slows convergence by 0.89×. Eva converges to the target average metric in 1,000 steps but is 1.69× slower than MKOR-H with 600 steps.Source: §3.4 · p. 43

ResNet-50 Experiments

• 1.49× speedup over SGD lets MKOR reach 75.9% ResNet-50 accuracy in 57 epochs versus 88, and MKOR remains 1.04× faster than KAISA’s 54 epochs.HyLo is not compared because it fails to reach the target, attaining 75.6% with tuning; MKOR’s advantage is less pronounced than in BERT because ResNet has smaller model dimension.Source: §3.4 · pp. 43–44, Figure 3.3 · p. 44

Inversion Frequency

• 10-range inversion frequency makes MKOR’s iteration cost nearly frequency-independent, so more frequent factor updates improve convergence without increasing per-iteration cost.The experiment uses a simple autoencoder on CIFAR-100; stale factors can lead to local minima, whereas MKOR’s frequent updates improve end-to-end training speed.Source: §3.4 · p. 44, Figure 3.4 · p. 45

Performance Analysis and Memory Overheads

• 1.5× lower memory overhead than KFAC/KAISA is achieved by MKOR, while its reduced factor-computation and inversion costs are especially beneficial for BERT-Large-Uncased.For ResNet-50, MKOR substantially reduces KAISA’s factor computation and inversion cost; for BERT, MKOR’s inversion cost is much smaller than total training time, enabling speedups.Source: §3.4 · pp. 44–45

• Rank-1 covariance approximations have low error across BERT-Large pretraining and ResNet-50, and higher-rank approximations are unnecessary in practice.Tests on well-known benchmarks found no model without low-rank covariance matrices.Source: §3.4 · p. 46, Figure 3.6 · p. 47
• Covariance rank is min(b, d), so small per-GPU batches directly produce low-rank matrices, while over-parameterization keeps them low-rank even with larger batches.Dependent activation and output-gradient features explain the persistent low-rank structure observed in larger-batch settings.Source: §3.4 · p. 46
• Higher-rank MKOR extensions can compute updates in O(rd^2), but the required SVD removes the main computational advantage of low-rank computation.The chapter reports no cheaper covariance approximation method beyond its rank-1 approach.Source: §3.4 · pp. 46, 48

3.5 Conclusion

• MKOR reduces inversion complexity from O(d^3) to O(d^2), lowers communication overhead to O(d), and delivers up to 2.57× faster training.It applies low-rank rank-1 updates and stability mechanisms for lower-precision communication while accelerating convergence.Source: §3.5 · p. 48

4.1 Introduction

• The introduction frames SLOPE as addressing the cost of all three training matrix multiplications by reducing FLOPs and memory traffic through hardware-supported N:M sparsity.Unstructured sparsity lacks hardware support, whereas semi-structured N:M sparsity balances model quality with acceleration via NVIDIA Sparse Tensor Cores.Source: §4.1 · p. 49
• SLOPE resolves N:M transposability by imposing a new N:M pattern after transposing the sparsified weights, enabling acceleration in both forward and backward passes.This avoids restrictive transposable masks, whose search can cause training slowdowns of up to 8.4×.Source: §4.1 · pp. 49–50
• SLOPE introduces lazy low-rank adapters as parallel computational paths only during the final 1% of pretraining, preserving sparse deployment while recovering accuracy.The adapters converge faster than the original parameters at the same parameter count and avoid dense fill-in of sparse weights.Source: §4.1 · p. 50

SLOPE: Double-Pruned Sparse Plus

• SLOPE: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs [101] combines double-pruned backward passes with low-rank adapters to jointly leverage the Trinity.The chapter’s publication is based on work presented at ICLR 2025 [101].Source: LLMs · p. 49, §4.1 · p. 50

4.2 Additional Related Work

• Prior pruning methods use unstructured patterns that current hardware cannot accelerate or incur substantial overhead when searching for structured masks [142] [166].Post-pretraining pruning can also be costly and often fails to maintain accuracy [36] [136].Source: §4.2 · p. 51
• Existing sparse-plus-low-rank work has not investigated sparse-plus-low-rank pretraining and does not provide acceleration in both fine-tuning directions.Earlier adapters are typically added at the beginning of fine-tuning, introducing extra overhead, although low-rank adapters reduce fine-tuning costs and enable task switching [61].Source: §4.2 · p. 51

4.3.1 Double-pruned Backward Pass

• Double-pruned N:M weight tensors accelerate both forward and backward computations while retaining convergence guarantees under the stated random-mask assumptions.The formulation prunes weights along both row and column dimensions, and dynamic column-mask changes do not detrimentally affect optimizer convergence.Source: §4.3.1 · pp. 52–53
• Double pruning increases sparsity by 12.5%, 9.375%, and 3.39% for 1:2, 2:4, and 2:8 patterns, respectively, with smaller gains as M increases.The reduced surplus of zeros at larger M is associated with fewer computational errors and greater computational robustness.Source: §4.3.1 · p. 53

4.3.2 Lazy Low-rank Adapters

• Lazy low-rank adapters add a compact correction to doubly-pruned weights, preserving sparse matrix-matrix efficiency while mitigating pruning-induced quality loss.The adapters are used exclusively during the final 1% of training iterations, reducing additional training cost while trading memory, computation, and quality through rank r.Source: §4.3.2 · pp. 53–54

4.3.3 Sparse Kernels

• Sparse kernels mask gradients for pruned weights and update weights and their transposes directly in sparse format, while supporting weight decay and half-precision gradient scaling.The implementation supplies sparse APIs for the forward pass, backward gradients, optimizer operations, and storage of sparse tensors.Source: §4.3.3 · p. 54

Algorithm 2 Accelerated Sparse Pretraining Algorithm for a Linear Layer

• Algorithm 2 initializes sparse weight representations and processes each mini-batch through sparse forward, backward, gradient, and optimizer-update operations.Its inputs are the weight, training set, weight decay α, and gradient scaling factor γ; its output is the updated weight.Source: §4.3.3 · p. 54, Algorithm 2 Accelerated Sparse… · p. 55
• Algorithm 2 applies sparse matrix multiplication in forward and backward passes, prunes and compresses weight gradients, then updates synchronized sparse weights and transposes.The accuracy experiments pretrain GPT2-Small and GPT2-Large on OpenWebText and evaluate MMLU, Arc Challenge, OpenBookQA, and validation perplexity.Source: Algorithm 2 Accelerated Sparse… · p. 55, §4.4.2 · p. 58

4.3.4 SLOPE Runtime Optimization

• 12% faster inference and 4% faster training result from tiling large upsample tensors into square matrices optimized for 2:4 sparse multiplication.Tiling addresses cuSPARSELt’s declining upsample speedup near hidden dimension 4000, and the products are concatenated afterward.Source: §4.3.4 · p. 55
• Up to 6% end-to-end inference-speed improvement comes from concatenating downsample tensors with sparse weights and using fused matrix-multiplication-plus-addition kernels.These optimizations reduce kernel calls, cache access, and low-arithmetic-intensity overhead from separate low-rank adapter computations.Source: §4.3.4 · pp. 55–56

4.4 Experimental Results

• SLOPE evaluates speedup and memory reduction on OPT models from 2.6B to 66B parameters, LLaMA-3-8B, and Mistral-v0.3-7B, while reporting medians across repeated runs.Accuracy evaluation is primarily limited to smaller models up to 774M parameters because full LLM pretraining is computationally expensive.Source: §4.4.1 · p. 56, §4.4 · p. 56

4.4.1 End-to-end Speedup and Memory Saving: Pretraining and Inference

• SLOPE achieves efficient and accurate models during both training and inference, unlike FST, which requires dense final iterations that remove inference-speed benefits.SLOPE prunes weights across MLP and self-attention modules without FST’s dense fine-tuning limitation.Source: §4.4.1 · p. 57
• 68% training-memory reduction and 54% inference-memory reduction demonstrate SLOPE’s substantial memory savings from sparse weights, indices, masks, gradients, and reduced optimizer states.The inference reduction stores only nonzero weights and indices, while training also benefits from reduced optimizer-state storage.Source: §4.4.1 · pp. 57–58

4.4.2 Pretraining Accuracy Results

• SLOPE achieves lower perplexity than Wanda [136] and Extended SR-STE while consistently exceeding Extended SR-STE in zero-shot accuracy.The comparison covers GPT2-Small and GPT2-Large validation perplexity and GPT2-Small zero-shot tasks.Source: §4.4.2 · pp. 58–59, Table 4.3 · p. 59
• Adding lazy low-rank adapters benefits static and dynamic sparse training by allocating updates away from weights that may later be pruned.Extended SR-STE can waste gradient updates on weights ultimately removed at inference.Source: §4.4.2 · p. 59
• Low-rank adapters recover pretraining accuracy when introduced only during the final 1% of iterations, with higher ranks improving performance but increasing computation.The ablation evaluates adapter ranks 4, 16, and 64 on SQuAD and GLUE.Source: §4.4.2 · p. 60, Table 4.4 · p. 60
• Adapters converge rapidly: the downsample adapter converges within 100 iterations, while 100-iteration training still yields comparable downstream-task results.The upsample adapter converges slightly more slowly.Source: §4.4.2 · p. 60, Figure 4.3 · p. 60
• Increasing sparsity reduces accuracy across all tasks; pruning the first 12 blocks causes significantly higher drops, especially on GLUE, while later blocks can be sparsified more aggressively, and SLOPE consistently outperforms Wanda.Source: §4.4.2 · p. 61

4.4.3 Ablation Studies

• Low-rank adapter speedups fall below ideal linear scaling at small ranks, but larger matrices narrow the gap through higher arithmetic intensity and better tensor-core utilization.The measured behavior reflects departures from the compute-bound assumption described by the Roofline model.Source: §4.4.3 · p. 61, Figure 4.4 · p. 62
• Implementation optimizations improve efficiency: fusing adapter operations raises inference speedup by up to 6%, while matrix tiling improves training speed by up to 4% and inference speed by up to 12%.The optimizations reduce kernel launches, increase arithmetic intensity, and divide upsample matrices into square tiles.Source: §4.4.3 · p. 62
• The experiments compare pruning only MLP modules with pruning both MLP and self-attention modules during Phase-2 BERT-Large-Uncased pretraining.Source: §4.4.3 · p. 63
• Sparsifying more modules produces a consistent but slight decrease in model quality across the evaluated SQuAD and GLUE settings.Source: §4.4.3 · p. 63
• The marginal performance change suggests SLOPE is relatively insensitive to which modules are pruned, maintaining competitive quality across diverse sparsity configurations.Source: §4.4.3 · p. 63

4.5 Conclusion

• SLOPE combines double-pruned backward passes with lazy low-rank adapters to accelerate N:M sparse pretraining while recovering model capacity.It achieves up to 1.25× pretraining and 1.54× inference speedups.Source: §4.5 · p. 63
• The transition from SLOPE to OPTIMA moves the Compression Trinity from pretraining efficiency toward inference on commodity hardware, where compounded compression error makes sparsity foundation quality critical.Sparsity is treated as structurally destructive because deleting connections can leave information unrecoverable through later quantization or low-rank adaptation.Source: §4.5 · pp. 63–64, §5.1 · p. 65

Reconstruction

• The paper’s OPTIMA chapter is based on “OPTIMA: Optimal One-Shot Pruning for LLMs via Quadratic Programming Reconstruction,” [98].The work was conducted with Samuel Kushnir, Amir Yazdanbakhsh, and Maryam Mehri Dehnavi.Source: Reconstruction · p. 65

5.1 Introduction

• OPTIMA formulates post-training weight reconstruction as an exact constrained convex quadratic program rather than a greedy approximation.The method targets a one-shot regime using only a small calibration dataset and static, layer-wise optimization.Source: §5.1 · pp. 65–66
• A shared Hessian H = X^T X lets OPTIMA decompose reconstruction across columns and guarantee per-column global optimality without additional weight assumptions.This formulation is provably equivalent to the least-squares objective after mask selection.Source: §5.1 · p. 66
• OPTIMA provides an accelerator-friendly pipeline that solves thousands of constrained column QPs in parallel and can run on a single accelerator.It accumulates one Hessian per layer, enforces fixed entries through bounds, and uses efficient matrix–vector products.Source: §5.1 · pp. 66–67
• OPTIMA acts as a drop-in weight-update step for Wanda, SparseGPT, and Thanos, consistently improving their accuracy without fine-tuning.This modularity separates mask selection from globally optimal reconstruction.Source: §5.1 · pp. 66–67
• Up to 3.97% absolute gains on six zero-shot downstream benchmarks show OPTIMA improves sparse models without post-pruning fine-tuning.The evaluation covers multiple model families and both unstructured and 2:4 semi-structured sparsity.Source: §5.1 · p. 66

5.2 Additional Related Work

• Prior pruning methods span layer-wise approaches such as OBS and end-to-end approaches such as OBD, but scaling second-order optimization remains difficult.SparseGPT [36], Thanos [64], Wanda [136], and related methods trade approximation, weight-update capability, or optimization cost for scalability.Source: §5.2 · pp. 67–68

5.3 Preliminaries

• Post-training pruning uses calibration data and layer-wise reconstruction, but optimal mask selection is NP-hard and direct weight updates are expensive for large LLMs.Existing heuristics avoid updates or modify only subsets of weights, trading optimality for computational efficiency.Source: §5.3 · p. 68

5.4 OPTIMA: Optimal Weight Updates via Quadratic Programming

• OPTIMA efficiently and optimally updates all remaining weights after selecting a pruning mask by solving independent quadratic programs in parallel on GPUs or TPUs.The approach addresses the costly weight-update step in post-training pruning.Source: §5.4 · p. 69

5.4.1 Reformulation as a Quadratic Program with Linear Constraints

• After fixing a mask, the Frobenius-norm reconstruction objective separates by weight columns, yielding independent constrained QPs with shared positive-semidefinite Hessians.Pruned entries impose linear zero constraints, while unpruned entries remain optimization variables, enabling accelerator parallelization.Source: §5.4.1 · p. 69

5.4.2 Reformulation as an Unconstrained Quadratic Program

• The unconstrained formulation eliminates equality constraints by substituting fixed pruned-weight changes, but produces heterogeneous column-wise problems that are harder to batch efficiently.OPTIMA therefore adopts the constrained formulation because its uniform structure better supports high-throughput GPU and TPU execution.Source: §5.4.2 · p. 70

Algorithm 3 Layer-wise Pruning with Batched Column-wise Quadratic Programming

• Algorithm 3 processes each LLM layer, estimates Hessians from calibration inputs, and applies batched column-wise QP updates to produce a pruned, updated model.Its inputs include a pre-trained LLM, calibration data, pruning masks, a QP solver, and batch size.Source: Algorithm 3 Layer-wise Pruning with… · p. 71
• The layer-wise pruning algorithm retrieves a mask, partitions weight columns into batches, processes each column, and returns the updated model and masks.The procedure is organized around batched column-wise quadratic-program updates.Source: Algorithm 3 Layer-wise Pruning with… · p. 71

5.4.3 Solving the Quadratic Programs

• OPTIMA uses rAPDHG, whose Hessian matrix-vector bottleneck suits GPUs and TPUs, with provably optimal linear convergence and a JAX implementation supporting parallel column-wise QPs.The shared Hessian and simple bounds make the constrained programs suitable for specialized solvers.Source: §5.4.3 · p. 71

5.4.4 Efficient Implementation

• Fixed-variable equality constraints are enforced through matching upper and lower bounds, avoiding prohibitively large explicit equality-constraint matrices.This formulation locks each constrained variable to its desired value without the associated matrix overhead.Source: §5.4.4 · p. 72
• OPTIMA enables pruning large LLMs on a single GPU by batching column-wise QPs while retaining parallel solution efficiency.Batching reduces memory overhead and still leverages parallel QP solving.Source: §5.4.4 · p. 72
• Incremental covariance accumulation avoids storing the full output tensor and produces the same Hessian estimate as direct computation, making the method scalable to very large LLMs.The covariance is accumulated sequence by sequence rather than materializing all outputs simultaneously.Source: §5.4.4 · p. 72

5.5 Experiments

• OPTIMA’s evaluation covers LLaMA, Gemma, and Qwen families using zero-shot downstream tasks and WikiText2 perplexity, with calibration from 128 C4 samples of 2048 tokens.The experiments use the Language Model Evaluation Harness and assess MMLU, Piqa, Arc-Easy, Arc-Challenge, WinoGrande, and OpenBookQA.Source: §5.5 · p. 72
• Up to 1.80% average accuracy improvement across six downstream tasks is achieved by OPTIMA at 50% unstructured sparsity on Gemma-3-1B.OPTIMA’s weight-update scheme consistently outperforms pruning methods using their original updates, including Wanda, SparseGPT, and Thanos.Source: §5.5 · pp. 73–74
• OPTIMA consistently outperforms other methods under 2:4 sparsity, achieving superior accuracy, with clear improvements for most models, particularly those in the 1.5B and 3B range.Pruning was applied only to MLP layers, producing 38% to 41% overall sparsity; with Wanda, 3B accuracy improved from 49.48% to 50.63%, while 1.5B improved from 46.01% to 47.26%.Source: §5.5 · p. 74
• OPTIMA delivers average accuracy gains of up to 2.53% across downstream tasks and increases Wanda’s average accuracy from 38.77% to 42.74%, preserving utility under extreme sparsity.Its QP formulation mitigates pruning error, leading to lower perplexity and higher downstream performance.Source: §5.5 · p. 74
• OPTIMA improves Qwen-2.5 accuracy across model sizes and mask selectors, including a 4.19% absolute gain for Wanda and a 1.53% gain for Thanos at 60% sparsity.At 50% sparsity on Qwen-2.5 3B, Wanda rises from 54.02% to 55.33% and SparseGPT from 54.70% to 55.69%; at 60%, Wanda rises from 43.67% to 47.86% and Thanos from 48.45% to 49.98%.Source: §5.5 · pp. 74–75
• ADAM can diverge or produce suboptimal updates, whereas OPTIMA’s convergent solver yields more reliable model quality, including 35.44 perplexity versus 205.82 for Wanda on OPT-125M.On Gemma 3 1B, Wanda+OPTIMA reaches 44.01% accuracy versus 43.72% for Wanda+ADAM, although SparseGPT+ADAM has slightly lower perplexity than OPTIMA in one case.Source: §5.5 · p. 75
• OPTIMA’s pruning time scales with model size, requiring approximately 2.5 h for LLaMA 3.2 1B and Gemma 3 1B, 5.5 h for Gemma 2 2B, 7.0 h for LLaMA 3.2 3B, and up to 40.0 h for LLaMA 3.1 8B.Similar pruning times across similarly sized models indicate robustness across diverse architectures.Source: §5.5 · p. 75

5.6 Conclusion and Limitations

• Up to 3.97 percentage points: OPTIMA improves zero-shot accuracy across LLM families without fine-tuning, including at sparsity levels ≥60%.It operates as a drop-in weight-update step for common mask selectors and uses globally optimal column-wise quadratic programs with shared-Hessian parallelism.Source: §5.6 · p. 76
• OPTIMA remains constrained by fixed, pre-determined masks that may misalign with information distribution, while sparsity alone can leave an accuracy gap from dense models.PATCH addresses mask flexibility through fine-tuning and learnable hybrid sparsity, while later work reintroduces low-rank adaptation to bridge the remaining gap.Source: §5.6 · p. 76

6.1 Statement of Contributions

• PATCH was a collaborative effort: Mohammad Mozaffari developed the concept and experiments, Younes Hourri formulated tile-level probability distributions and led pruning and hardware acceleration, and both authors revised the manuscript.Maryam Mehri Dehnavi supervised the work.Source: §6.1 · p. 84

6.2 Introduction

• PATCH addresses the accuracy ceiling caused by rigid masks through flexible tile-level sparsity and joint optimization of within-2:4-tile sparsity patterns and tile-level configurations.Source: §6.2 · pp. 84–85

6.3.1 Pruning methods

• Unstructured sparsity offers flexible allocation but irregular memory access, whereas structured 2:4 sparsity accelerates hardware but enforces uniform 50% sparsity and can reduce accuracy.The 2:4 pattern requires at least two zeros in every four contiguous weights and fails to account for differing layer sensitivities.Source: §6.2 · p. 85

6.5 PATCH

• 1.18×–1.38× end-to-end speedup over the dense baseline, while improving accuracy by 0.37%–2.96% versus MaskLLM, on LLaMA-2 7B using an A6000 GPU.PATCH assigns tiles to either dense or 2:4 sparse computation, enabling global sparsity ratios between 0% and 50% while remaining compatible with acceleration libraries.Source: §6.2 · p. 85

6.3.2 Complementary compression techniques

• Quantization, low-rank adaptation, and knowledge distillation are complementary to pruning and can support high compression ratios without sacrificing accuracy.Quantization reduces parameter and activation precision, while low-rank factors reduce parameter counts while maintaining expressivity.Source: §6.3.2 · pp. 86–87

6.4 Preliminaries

• Gumbel Softmax provides differentiable categorical mask sampling, with lower temperatures producing more peaked distributions that approach one-hot selections.MaskLLM applies this mechanism to learn among six possible 2:4 patterns while keeping the underlying weights fixed.Source: §6.4 · p. 87

6.5 PATCH

• PATCH learns masks by probabilistically selecting dense or sparse tiles and jointly choosing 2:4 patterns while keeping model weights frozen.Gumbel–Softmax logits explore tile configurations and converge toward masks balancing accuracy and sparsity; the tile mask interpolates between dense and 2:4 behavior.Source: §6.5 · p. 88
• PATCH enables a flexible 0%–50% global sparsity ratio by controlling how many tiles remain dense rather than enforcing a fixed 2:4 pattern everywhere.Its regularizer penalizes deviations from the target sparsity, while global allocation prunes sensitive linear layers less aggressively and removes redundant elements.Source: §6.5 · p. 88

Algorithm 4 Joint Tile & 2:4 Mask Learning

• PATCH’s objective combines modeling loss, target-density sparsity regularization, and weight regularization, with annealed Gumbel–Softmax producing discrete masks for inference.Positive tile logits select dense tiles and negative logits select sparse tiles after training.Source: Algorithm 4 Joint Tile & 2:4 Mask… · p. 89
• PATCHTile freezes sparse-mask parameters and optimizes only tile decisions, reducing learnable parameters but limiting mask flexibility and performance under memory constraints.This variant makes training feasible for an 8B model on a single 80GB GPU, whereas PATCHJoint performs joint optimization.Source: Algorithm 4 Joint Tile & 2:4 Mask… · pp. 89–90

6.6 Efficient deployment of PATCH

• STOICC extends Triton with hybrid tile-level sparsity, addressing existing tools’ dense-only or fixed 2:4 limits and providing a suitable backend for accelerating PATCH.Its inspector benchmarks candidate kernel configurations.Source: §6.6 · p. 90

6.7.1 Model Quality Results

• PATCHJoint jointly optimizes dense-tile locations and within-tile sparsity patterns, providing a continuous sparsity–quality tradeoff for smaller models.PATCHTile freezes the fine-grained 2:4 structure while optimizing dense-tile locations for larger models, retaining a comparable flexible tradeoff.Source: §6.7.1 · pp. 91–92
• PATCH consistently surpasses Wanda, SparseGPT, and magnitude pruning, while flexible sparsity from 45% to 25% progressively aligns accuracy and perplexity with dense models.PATCH improves over fixed-ratio MaskLLM by allocating dense tiles flexibly; unstructured sparsity remains better, but the hybrid gap is not significant.Source: Table 6.2 · p. 91, §6.7.1 · p. 92

6.7.2 Understanding the components of PATCH

• Tile size has limited effect on model quality, although smaller tiles may reduce hardware efficiency and therefore require hardware-aware selection.The 4 × 4 configuration maximizes quality in the tested setup, while larger tiles show minimal variation.Source: §6.7.2 · pp. 92–93, Table 6.5 · p. 93
• Joint optimization of tile and 2:4 masks consistently outperforms tile-only training, while tile-only training remains practical for larger, resource-constrained models.Among frozen masks, MaskLLM is strongest; one-shot methods diverge at lower sparsity, with SparseGPT best overall in that comparison.Source: §6.7.2 · p. 93, Table 6.7 · p. 94
• Global sparsity improves quality by pruning redundant middle layers more aggressively while preserving capacity in sensitive deeper and initial blocks.Within blocks, MLP up, gate, and down matrices absorb most pruning; key and value matrices are never pruned, while query matrices receive the most.Source: §6.7.2 · pp. 93–94
• PATCH assigns most sparsity to redundant MLP matrices, while preserving key and value matrices largely dense and pruning query matrices most aggressively within attention.The observed allocation is consistent across evaluated models and reflects greater sensitivity in self-attention than in MLP components.Source: §6.7.2 · p. 94

6.7.3 Speedup and memory savings

• PATCH inference reaches 1.18×, 1.27×, and 1.38× throughput over dense models at 25%, 35%, and 45% sparsity, while memory falls to 0.76×, 0.68×, and 0.59%.These measurements use LLaMA-2 7B, STOICC, batch size 16, and an A6000 GPU.Source: §6.7.3 · p. 94

6.8 Conclusion and Limitations

• PATCH uses tile-level dense-or-2:4-sparse masks to learn adaptive sparsity ratios from 0% to 50%, improving accuracy over static 2:4 methods with up to 1.38× speedup.Experiments cover models up to 8B parameters and position PATCH as practical hybrid sparsity for efficient inference.Source: §6.8 · p. 95
• Unlike OPTIMA’s zero-training regime, PATCH requires fine-tuning because learnable masking adds overhead, whereas SLIM integrates quantization and low-rank approximation to overcome sparsity’s accuracy wall.The section frames PATCH and OPTIMA as complementary deployment solutions and motivates SLIM as the thesis’s unified Compression Trinity framework.Source: §6.8 · p. 95

7.1 Introduction

• SLIM resolves compounded sparsity–quantization error by combining hardware-friendly sparsity, uniform quantization, and mathematically derived one-shot low-rank error correction.SLIM-Quant can optimize quantization parameters, while SLIM-LoRA analytically computes adapter values without retraining; the sparsity stage can use masks from Wanda [136], OPTIMA, PATCH, or future selectors.Source: §7.1 · pp. 98–99
• SLIM improves average accuracy by 5.66% on LLaMA-2-7B under 2:4 sparsity and 4-bit quantization, while reaching up to 0.6% higher accuracy than dense models at equal parameter budgets.It also improves accuracy by up to 0.5% at equal total bit budgets and achieves up to 3.78× and 3.75× layer-wise speedups on RTX3060 and A100 GPUs, respectively.Source: §7.1 · p. 99

7.2.1 Pruning

• Prior pruning methods remove redundant weights to reduce inference cost, but second-order approaches face prohibitive computation or poor scaling for LLMs.OBD uses second-order loss information, while OBS [54] incurs cubic Hessian-inversion costs; WoodFisher [134], SparseGPT, OBC [35], and Wanda address these limitations with approximations or lightweight criteria.Source: §7.2.1 · pp. 99–100

7.2.2 Quantization

• Quantization is challenged by outliers: group quantization [47] reduces this sensitivity, whereas joint pruning-and-quantization methods such as JSQ struggle at low bit widths.Representative alternatives include OPTQ, AWQ, SmoothQuant [155], OmniQuant, AffineQuant, and QuaRot.Source: §7.2.2 · p. 100

7.2.3 Low-rank Adapters

• Adapter-based methods recover accuracy lost during quantization and address sparsity, but some require hundreds of millions of fine-tuning tokens and sparse adapters increase inference overhead.Source: §7.2.3 · p. 100

7.2.4 Sparse Plus Low-Rank Matrix Decomposition

• Classical matrix decomposition separates a matrix into sparse and low-rank components; RPCA provides exact recovery of both under mild incoherence conditions, while Stable PCP remains robust in noisy settings.OATS [164] formulates post-training weight compression.Source: §7.2.4 · pp. 100–101
• Sparse-plus-low-rank decomposition has yielded better accuracy–compression trade-offs than pure pruning or pure low-rank factorization, motivating SLIM’s formulation W ≈ W_C + LR.Unlike classical RPCA, SLIM uses a quantized sparse component and a sequential quantize-then-sparsify-then-correct pipeline rather than jointly optimizing both components.Source: §7.2.4 · p. 101

7.3 Preliminaries

• Model compression commonly minimizes layer-wise output discrepancies because directly optimizing whole-model differences is computationally infeasible.OBS provides the foundational layer-wise formulation using calibration data, while symmetric quantization is sensitive to scale selection and outliers.Source: §7.3 · pp. 101–102

7.4 Quantized sparse plus low-rank approximation of LLMs

• SLIM integrates quantization, pruning, and saliency-based low-rank adapters in a pipeline that reduces the accuracy loss from both compression sources.It first applies SLIM-Quant, then prunes the quantized weights, and finally adds low-rank adapters to correct quantization and pruning errors.Source: §7.4 · pp. 102–103

7.4.1 SLIM-Quant quantization method

• SLIM-Quant uses symmetric quantization with a data-driven scaling parameter because standard weight distributions do not accurately match tested candidate PDFs.It numerically integrates the observed weight histogram and refines a multigrid search around the minimum-error region.Source: §7.4.1 · pp. 103–104

7.4.2 SLIM-LoRA low-rank adapters

• SLIM-LoRA extends sparse-plus-low-rank decomposition from Robust PCA and prior neural-network and LLM compression work [164] to joint compression.Unlike solving a single convex program, SLIM derives the low-rank component analytically from compression error with saliency weighting.Source: §7.4.2 · p. 104
• SLIM derives low-rank adapters analytically from compression error using saliency-weighted SVD, enabling one-shot recovery without iterative optimization.The formulation models quantization and pruning as additive errors and uses an invertible, additive saliency function to recover adapters.Source: §7.4.2 · pp. 104–105

7.4.3 Low-rank adapter quantization

• SLIM quantizes low-rank adapters with 4-bit AbsMax group quantization, reducing adapter overhead by 4× while retaining their compression and performance contribution.Groups of 128 elements share a quantization parameter to accommodate the adapters’ long-tailed distributions.Source: §7.4.3 · p. 106

7.4.4 Optional Post-Compression Fine-Tuning

• Post-compression fine-tuning is presented as challenging because traditional methods’ parameter and memory demands are computationally prohibitive.This motivates considering an optional fine-tuning stage after compression, but the supplied passage provides no further outcome.Source: §7.4.4 · p. 106
• SLIM uses parameter-efficient low-rank adapters as the only tunable components, freezing sparse and quantized weights during optional fine-tuning.This limits optimization to a small subset of parameters, preserving compression benefits while enabling adaptation to new data or tasks.Source: §7.4.4 · pp. 106–107
• SLIM supports quantization-aware adapter training with a straight-through estimator and reduces overhead using custom Triton quantization and dequantization kernels.The compressed base weights remain frozen while adapters can be quantized during fine-tuning.Source: §7.4.4 · p. 106
• Fine-tuning requires only a short phase over thousands of tokens, rapidly improving accuracy while significantly reducing computational requirements.The motivation is that conventional optimizers can add 4× memory overhead for gradients and optimizer states, making parameter-efficient alternatives important for compressed models.Source: §7.4.4 · pp. 106–107

Accuracy results

• SLIM outperforms competing pruning and quantization approaches across 2:4 and unstructured sparsity benchmarks with 50% sparsity and 4-bit weight quantization.Table 7.1 reports average zero-shot accuracy for OPT and LLaMA-2 models; SparseGPT and Group OPTQ are also competitive.Source: §7.5 · p. 107, Table 7.1 · p. 108
• Several baselines have practical limitations: OmniQuant and AffineQuant often produce NaNs on OPT-350M, while multiple methods encounter out-of-memory errors on a single A100-40GB GPU.JSQ performs well for LLaMA-2 but has difficulty compressing the OPT family, limiting broader applicability.Source: §7.5 · p. 108
• SLIM-LoRA improves over Naive-LoRA through saliency-aware adapter design, while SLIM-LoRAQ reduces adapter overhead with negligible accuracy change.Lightweight fine-tuning further improves both methods, with larger gains for SLIM-LoRA because of its saliency-aware design.Source: §7.5 · p. 108

Integration with Weight Update

• Combining OPTIMA weight updates with SLIM-LoRA further boosts accuracy when applying the compression trinity.The optimal weight values are obtained by solving a calibration-data quadratic program after determining the low-rank adapter values.Source: §7.5 · pp. 108–109, Table 7.3 · p. 109

Integration with Hybrid Sparsity

• Integrating PATCH masks with SLIM-LoRA and Group AbsMax quantization enables controllable tradeoffs between compression ratio and model quality.The hybrid-sparsity experiments evaluate LLaMA-2 7B and LLaMA-3.1 8B, demonstrating that PATCH’s flexible sparsity transfers to joint compression.Source: §7.5 · p. 109, Table 7.4 · p. 110

Comparison of large compressed and small dense models

• SLIM-LoRAQ, with or without fine-tuning, consistently outperforms dense models and other compression methods at the same parameter size.The comparison focuses on 2:4 sparsity in the OPT family and plots average accuracy against parameter size.Source: §7.5 · p. 110

Speedup

• SLIM achieves inference speedups through optimized sparse and quantized matrix multiplication, with larger matrices in self-attention and feed-forward modules yielding greater gains.Experiments use Sparse Marlin integrated with vLLM on NVIDIA RTX 3060 GPUs, with A100 results provided separately.Source: §7.5 · p. 110

• At equal parameter size, SLIM outperforms dense models and other compression techniques.Source: Figure 7.2 · p. 111

Sparsity results

• Magnitude Pruning performs worst, while Wanda and SparseGPT achieve comparable sparse-model accuracy, with larger gaps under semi-structured sparsity.Table 7.5 reports average zero-shot accuracy for LLaMA-2 and OPT models without quantization.Source: §7.5 · p. 111, Table 7.5 · p. 112
• Low-rank adapters improve accuracy, with saliency-based SLIM approximations performing strongly and brief fine-tuning further boosting low-rank results.Source: §7.5 · p. 111

Quantization-only results

• Adding low-rank adapters to Group AbsMax significantly boosts accuracy, while SLIM variants match or exceed Group AbsMax with adapters and lightweight fine-tuning reaches state-of-the-art accuracy.These quantization experiments disable sparsity and compare SLIM-Quant with Group AbsMax, OPTQ, AWQ, OmniQuant, AffineQuant, and L2QER.Source: §7.5 · p. 112, Table 7.6 · p. 113

Additional experiments

• SLIM’s accuracy trends hold on WikiText-2, while fine-tuning cost falls from over 36 days for 13B-parameter models to 14 hours on one GPU.Additional analyses also cover speedup, computation and memory reduction, compression cost, adapter rank, sparsity ratio, and calibration sample count.Source: §7.5 · p. 113
• 50% sparsity combined with 4-bit quantization achieves better compression than using 2-bit quantization alone.Source: §7.5 · p. 113

8.1 Summary of Contributions

• The Compression Trinity is presented as a lifecycle-wide framework that jointly targets compute FLOPs, memory bandwidth, and parameter redundancy rather than only post-training compression.The thesis spans training dynamics, static and dynamic sparsity, and unified one-shot deployment.Source: §8.1 · p. 116

8.1.1 The Trinity in Training Dynamics

• MKOR reduces curvature-update complexity from O(d^3) to O(d^2) and accelerates convergence by up to 2.57× over first-order baselines and 1.75× over KFAC.It combines block-diagonal sparsity, rank-1 low-rank updates, and quantization for second-order optimization.Source: §8.1.1 · p. 116

8.1.2 The Trinity in Post-Training and Inference

• OPTIMA stabilizes aggressive compression through column-wise quadratic programs with a shared Hessian, while PATCH enables adaptive tile-level sparsity ratios from 0–50%.Together, these methods address compounded quantization error and the rigidity of fixed hardware sparsity patterns.Source: §8.1.2 · p. 117
• Up to 5.66% accuracy improvement over state-of-the-art methods positions SLIM as a unified compressed model that can also outperform dense models at equivalent parameter budgets.SLIM integrates aggressive quantization, semi-structured sparsity, and saliency-based low-rank adapters to shift the efficiency Pareto frontier.Source: §8.1.2 · p. 117

8.2.1 BEAM: Blockwise Error Minimization for One-shot Compression of LLMs

• BEAM recovers up to 4.34% accuracy on sparse and quantized models using a single GPU in under four hours without end-to-end retraining.It minimizes feature-reconstruction error blockwise against the original model and serves as a compression-type-agnostic refinement stage.Source: §8.2.1 · p. 117

8.2.2 LEAP: Learnable End-to-End Adaptive Pruning of LLMs

• LEAP makes unstructured sparsity differentiable and learnable end-to-end, showing that optimal sparsity can shift during training as the model specializes.It treats parameter inclusion as a latent variable, relaxes masks to continuous probabilities, and uses straight-through estimation.Source: §8.2.2 · p. 118

8.2.3 SLICE: Selecting Layer-wise Configurations for Matryoshka-Style LLMs

• SLICE trains one super-model for multiple efficiency tiers, enabling runtime expansion or reduction of layers and precision as resource availability changes.Its nested configurations extend Matryoshka Representation Learning to compression itself, including a 2-bit core within a 4-bit shell.Source: §8.2.3 · p. 118

8.3 Limitations of the Current Approach

• The current methods have limited portability because PATCH, SLOPE, and SLIM rely on NVIDIA Sparse Tensor Core architecture and 2:4 semi-structured sparsity.The thesis identifies hardware coupling as a constraint on immediate generalizability and ease of adoption.Source: §8.3 · p. 118
• The approach remains limited by hardware-specific transferability, custom CUDA/Triton portability barriers, second-order optimizer tuning sensitivity, and substantial pipeline complexity.These constraints can offset speed gains and hinder plug-and-play adoption compared with simpler quantization methods.Source: §8.3 · pp. 118–119
• Aggregate benchmark accuracy may conceal disproportionate degradation for low-resource languages and minority groups, so disaggregated fairness audits are essential before high-stakes deployment.The thesis also leaves trillion-parameter scaling unexplored, where emergent behaviors and cross-node communication may change compression effectiveness.Source: §8.3 · p. 119

8.4 Future Research Directions

• Future work should extend the Trinity to million-token context windows by compressing KV caches and attention through quantization, sparsity, and low-rank approximations.The proposed direction targets attention and KV-cache bottlenecks as sequence lengths grow.Source: §8.4 · p. 119
• Hardware–algorithm co-design should eliminate sparse-index metadata overhead through deterministic or predicted patterns, while learnable activation masks could accelerate prefill without full retraining.These directions address sparse-format storage costs and the remaining activation-tensor bottleneck.Source: §8.4 · p. 119

8.5 Closing Remarks

• Jointly applying sparsity, quantization, and low-rank approximations integrates the Compression Trinity across LLM training and inference, enabling models dense in knowledge yet sparse in computation.The closing remarks frame efficiency as a design constraint and associate the lifecycle integration with MKOR and SLIM.Source: §8.5 · pp. 119–120

A.1 GLUE Results

• The appendix evaluates MKOR on GLUE classification tasks using BERT-Large-Uncased and provides the per-task results in Table A.1.The surrounding text identifies Appendix A.1 as the source of the GLUE results and Table A.1 as reporting metrics for the different tasks.Source: §A.1 · p. 132, Table A.1 · p. 132

A.2 Derivation of NGD Approximations

• The appendix derives NGD, KFAC, and SNGD approximations for second-order optimization, including Kronecker-factor and Sherman-Morrison-Woodbury-based inverse computations.KFAC factorizes the Fisher Information Matrix into two matrices under an independence approximation, while SNGD inverts a batch-sized kernel matrix with damping.Source: §A.2 · pp. 132, 134

A.3 Numerical Instability of Second-order Methods / A.4 Sensitivity to Learning Rate

• MKOR avoids numerical instability during matrix inversion and remains less sensitive to learning-rate choice than competing optimizers.KFAC factors can retain large condition numbers despite damping and eigenvalue filtering, whereas MKOR’s inversion complexity is independent of the condition number; Table A.2 reports its broad learning-rate robustness.Source: §A.3 · p. 134, §A.4 · p. 135

A.5 Scalability

• MKOR scales strongly on BERT-Large-Uncased across as many as 64 GPUs.The scalability evidence is presented in Figure A.3.Source: §A.5 · p. 135

A.6 Decaying Eigenvalues and Rank-1 Approximations

• As training converges, factor eigenvalues decay, making rank-1 approximations more effective and improving MKOR’s reliability after initialization.MKOR initializes factors with the identity, improving robustness to approximation noise in early iterations; later, training samples shape the factors more strongly.Source: §A.6 · pp. 135–136

A.7 Training Accuracy Experiments

• MKOR accelerates training by 1.26×, 1.31×, and 1.58× over HyLo-KIS, SGD, and KAISA on AlexNet trained on CIFAR-100.KAISA’s convergence was impaired by the small learning rates required to avoid exploding gradients.Source: §A.7 · p. 136
• MKOR achieves 1.22× and 1.43× speedups over SGD and HyLo-KIS on BERT-Large fine-tuned on IMDB, and 1.26× and 1.56× over SGD and HyLo on BERT-Base fine-tuned on SQuAD.KAISA fails to converge across the reported BERT experiments because of exploding gradients.Source: §A.7 · p. 136

A.8 Knee-Point Learning Rate Scheduler / Appendix A.7.

• The knee-point scheduler monitors accuracy improvement or loss reduction and lowers the learning rate when it detects a knee point.It is designed to avoid manually choosing when to reduce the learning rate, which is necessary for convergence but not known in advance.Source: §A.8 · p. 136, Appendix A.7 · p. 136

Appendix A.7.

• The appendix defines knee points using an accuracy/loss-rate threshold based on an exponential moving average and a selectable tolerance hyperparameter β.The knee-point learning rate is not used in the main-chapter experiments.Source: Appendix A.7 · pp. 136–137

A.9 Proofs

• The proofs establish that the optimizer’s matrix update remains positive-definite and that vector-vector, vector-matrix, and outer-product quantization errors scale as O(2ϵm^2d), O(2ϵm^2d), and O(2ϵm), respectively.The analysis also derives an O(4m^3d^2) error for the outer-product term Jjj^T J^T.Source: §A.9 · pp. 137–138

Supplementary Material for SLOPE

• The supplementary SLOPE material covers SR-STE comparison, initialization overhead, BERT-Large pretraining and downstream evaluation, bidirectional-mask overhead, sparsity-ratio and pruning-matrix analyses, implementation, GLUE, Flash Attention, dense-model comparison, and proofs.These topics are organized across Appendices B.1–B.10.Source: Supplementary Material for SLOPE · p. 143

B.1 Comparison with Dynamic Sparsity: SR-STE

• For the same training budget, SLOPE achieves lower perplexity than SR-STE because its static mask avoids computation spent updating weights that are ultimately pruned.SR-STE’s mask difference decreases toward a converged sparsity pattern, but the dynamic approach wastes computation during this transition; SLOPE fixes the mask at the first iteration.Source: §B.1 · p. 143, Figure B.1 · p. 144

B.2 cuSPARSELt Initialization Overhead: Static vs. Dynamic Sparsity

• cuSPARSELt setup time substantially exceeds multiplication time; static SLOPE masks amortize setup once, whereas dynamic masks can reduce speedups or cause slowdowns.Setup initializes handles, compresses 2:4 matrices, and generates metadata used by sparse matrix multiplication.Source: §B.2 · p. 144, Figure B.2 · p. 145

B.3 BERT-Large-Uncased: Pretraining and Downstream Evaluation

• Higher sparsity ratios generally increase BERT-Large-Uncased training loss, but loss or perplexity gaps do not directly predict downstream accuracy drops.Pretraining uses 7,038 iterations in phase 1 and 1,563 in phase 2, with sequence lengths 128 and 512, respectively.Source: §B.3 · pp. 144–145, Figure B.3 · p. 146

B.4 Performance overhead of bidirectional mask

• Bidirectional Masks incur significant runtime slowdowns versus dense baselines because of mask-search and associated overheads.The runtime comparison is reported in Table B.1 using the repository and models from the original method’s evaluation.Source: §B.4 · p. 145, Table B.1 · p. 145

B.5 Sparsity ratio analysis of double-pruned backward pass

• Double-pruned backward passes impose sparsity ratios generally below the original matrix density, with smaller N/M ratios producing lower imposed sparsity.The imposed ratio is derived from row-wise and column-wise pruning in the backward pass.Source: §B.5 · p. 146, Figure B.4 · p. 147

B.6 Sensitivity to the choice of pruning matrix

• Static pruning consistently yields lower perplexity than dynamic pruning, while pruning weights outperforms pruning inputs; pruning output gradients causes divergence.Dynamic pruning recomputes and stores dense matrix values before pruning at every iteration.Source: §B.6 · p. 147, Figure B.5 · p. 148

B.7 Implementation details

• Custom CUDA kernels support sparse setup, gradient masking, same-pattern matrix addition, and sparse-weight updates, reducing redundant gradient storage and enabling cuSPARSELt integration.The implementation also uses cuSPARSELt’s automated kernel search, whose setup cost is mitigated by repetitive matrix multiplications.Source: §B.7 · pp. 147–148

B.8 Task-specific GLUE results / B.9 Integration with Flash Attention

• SLOPE integrates orthogonally with FlashAttention-2: each improves training and inference speed, and combining them increases speedup further.Task-specific GLUE results are provided for all training settings in Table B.2.Source: §B.9 · p. 149, Table B.3 · p. 149, §B.8 · p. 149

B.10 Comparison with dense models

• SLOPE outperforms GPT2-Half on average, while SR-STE performs worse; adding low-rank adapters improves accuracy across all sparse pretraining methods.The comparison uses GPT2-Half, a model with half the transformer blocks of GPT2-Small, across zero-shot downstream tasks.Source: §B.10 · p. 150

B.11 Zero-shot GLUE results for GPT

• SLOPE outperforms SR-STE and GPT2-Half on average on zero-shot GLUE tasks, while SR-STE outperforms GPT2-Half.Table B.5 compares GPT models using different sparsity methods and LoRA ranks; E-SR-STE denotes Extended SR-STE.Source: §B.11 · p. 150, Table B.5 · p. 150

B.12 Extended SR-STE and FST implementation details

• Quality comparisons between SLoPe and FST are imbalanced because FST produces a dense model with approximately 2× more parameters than SLOPE’s sparse model with lightweight low-rank adapters.FST uses sparse pretraining followed by dense finetuning, whereas SLOPE retains sparsity and adds lightweight low-rank adapters.Source: §B.12 · p. 151
• SLoPe and Extended SR-STE are compared at equal parameter counts to isolate static versus dynamic masking in an iso-params model-quality comparison.The methods are compared separately for training speedups and memory savings, where model-quality comparison is less meaningful because their final parameter counts differ.Source: §B.12 · p. 151
• The appendix provides FST and Extended SR-STE implementations, with Extended SR-STE storing dense weights and pruning them on the fly.The listings define custom forward and backward passes for both algorithms.Source: §B.12 · pp. 151–152

B.13 Comparison of Depth and Width Pruning

• Preliminary retraining loss curves show no significant difference between depth pruning and width pruning during pretraining, with depth pruning sometimes outperforming width pruning.The comparison covers LLaMA-2-7B and Gemma-2-2B/9B; configurations are summarized in Tables B.7–B.9.Source: §B.13 · pp. 152–153, Figure B.6 · p. 156

Theorem 4.3.1

• The N:M sparsity analysis models added nonzeros and extra sparsity with binomial random variables under a uniform N:M mask.The derivation defines the success probability as N/M and obtains the expected added sparsity from the expected number of added nonzeros.Source: Theorem 4.3.1 · pp. 153–154

Theorem 4.3.2

• Theorem 4.3.2 derives an expected masked-gradient result using a uniformly random mask, the expectation matrix O ≜ E[M], and linearity of differentiation and expectation.The proof formulates the optimization target and assumes each mask entry equals one with probability N/M.Source: Theorem 4.3.2 · pp. 154–155

C.1 Calibration dataset size sensitivity

• OPTIMA shows consistent, though not significant, model-quality improvement as calibration samples increase, indicating robustness to calibration-dataset size.Unlike Wanda and SparseGPT, OPTIMA does not exhibit stochastic behavior with increasing sample counts.Source: §C.1 · p. 157, Figure C.1 · p. 158

D.1 STOICC Integration

• STOICC extends Triton with sparse code generation so PATCH can execute dense and sparse tiles within one mixed matrix operation.Its inspector autotunes tile sizes and execution schedules, tuning decoding first and reusing the selected decoding tile size for prefill.Source: §D.1 · p. 159
• 1.18×–1.38× end-to-end acceleration over the dense baseline is achieved by PATCH’s heterogeneous mixed-sparsity execution on LLaMA-2 7B at 25%–45% sparsity.Measurements use batch size 16 on an A6000 GPU, with additional timings reported on A100.Source: §D.1 · p. 160

D.2 Per Task Results

• The per-task appendix covers eight zero-shot benchmarks and reports PATCHJoint or PATCHTile against multiple pruning baselines at 25%–50% sparsity configurations.Tables D.3–D.7 provide task-level results for Qwen-2.5 0.5B, LLaMA-2 7B, LLaMA-3.1 8B, LLaMA-3.2 1B, and Gemma-3 1B.Source: §D.2 · p. 161
• PATCH maintains robust per-task accuracy across diverse zero-shot tasks and high sparsity levels compared with baseline pruning methods.The appendix reports PATCHJoint for smaller models and PATCHTile with fixed 2:4 sparsity for larger models.Source: §D.2 · p. 161
• PATCHJoint and PATCHTile maintain high accuracy across diverse tasks while preserving hardware-friendly sparsity patterns.The reported per-task results cover eight zero-shot tasks for Qwen-2.5 0.5B, LLaMA-2 7B, LLaMA-3.1 8B, LLaMA-3.2 1B, and Gemma-3 1B.Source: §D.2 · p. 162, Table D.3 · p. 162, Table D.4 · p. 162

D.3 Tile Transfer Learning

• Tile-prior initialization has little effect on final performance, with all priors producing nearly identical perplexity and random initialization often performing best.The global sparsity target can dynamically reallocate sparsity across layers during training, overriding fixed initialization effects.Source: §D.3 · p. 164, Table D.8 · p. 164

Supplementary Material for SLIM

• The SLIM supplementary material covers input quantization, fine-tuning, language modeling, sparse and quantized accuracy, speedups, costs, theoretical reductions, rank choices, calibration sensitivity, sparsity ratios, and group quantization.These analyses are organized across Appendices E.1–E.16, with the supplied overview identifying the listed topics.Source: Supplementary Material for SLIM · p. 165

E.2 Input Quantization

• Input quantization causes minimal accuracy loss for SLIM, demonstrating robustness with 4-bit weights and 8-bit inputs.The setup uses AbsMax 8-bit input quantization and FP8 weight quantization, selecting E4M3 or E5M2 according to the tensor’s maximum value.Source: §E.2 · pp. 165–166, Table E.2 · p. 166

E.3 Additional fine-tuning results

• Lightweight low-rank-adapter fine-tuning improves all baselines, with SLIM-LoRA achieving the largest gains through its saliency-based design.Table E.3 compares fine-tuned and non-fine-tuned OPT and LLaMA-2 models at 50% sparsity and 4-bit weight quantization.Source: §E.3 · p. 166, Table E.3 · p. 167

E.6 Sparsity vs. quantization

• SLIM achieves superior performance across language-modeling settings, while combining sparsity and quantization outperforms quantization alone at lower bitwidth.WikiText-2 experiments cover 2:4 and unstructured sparsity, input quantization, pruning-only, and quantization-only models; zero-shot accuracy and perplexity comparisons are reported in Tables E.10 and E.11.Source: §E.4 · p. 166, §E.6 · p. 171, Table E.10 · p. 171

E.7 Additional speedup results

• SLIM speedups scale favorably to larger models on A100-40GB GPUs, including LLaMA-3.1-405B, but Sparse Marlin can slow LLaMA-2-7B.The slowdown is attributed to Sparse Marlin’s sparse quantized matrix-multiplication kernels on A100 GPUs and is not specific to SLIM.Source: §E.7 · p. 172, Figure E.1 · p. 173

E.8 Fine-tuning costs

• Low-rank adapters reduce fine-tuning memory and computation by freezing sparse quantized weights, whereas direct fine-tuning becomes prohibitively expensive.Without adapters, fine-tuning took 12 hours for 125M models and over 36 days for 13B models; adapter-based fine-tuning used 300,000 C4 tokens on one H100 GPU.Source: §E.8 · pp. 173–174, Table E.12 · p. 174

E.9 Memory reduction analysis

• SLIM combines pruning, quantization, and low-rank adapters, with quantized adapters available to reduce adapter overhead.The memory analysis derives a reduction ratio under hidden dimension d, transformer-block count n, vocabulary size V, projection ratio a, and adapter rank r; Naive-LoRA and SLIM-LoRA use r = 0.1.Source: §E.9 · p. 174

E.10 Computation reduction analysis

• SLIM reduces inference FLOPs, and the overhead of rank r = 0.1 adapters in SLIM-LoRA and Naive-LoRA is not significant.The computation analysis distinguishes memory reduction from computation reduction, noting that quantization alone does not reduce floating-point inference computation.Source: §E.10 · p. 175, Table E.14 · p. 175

E.11 Compression costs

• Compression methods have similar GPU memory usage when processing one layer at a time, while matrix-multiplication methods such as Wanda are faster.The supplied passage begins the comparison of compression costs but does not report a complete set of method-level results.Source: §E.11 · p. 175
• Low-rank-adapter methods generally require longer compression times, while SparseGPT has comparable complexity despite performing only pruning and quantization.Adding adapters requires singular value decomposition, but SLIM’s saliency-based approach adds no significant overhead compared with Wanda-SVD.Source: §E.11 · p. 175

E.12 Rank analysis

• A rank of r = 0.1 substantially improves accuracy without significant inference overhead, while larger ranks further improve accuracy but increase computational and memory costs.Low-rank adapter overhead becomes negligible when r ≪1.Source: §E.12 · p. 176

E.13 Effects of calibration sample count

• SLIM demonstrates low sensitivity to calibration-sample count, remaining effective even when calibration data are limited.The analysis varies calibration samples from the C4 dataset for LLaMA-2-7B.Source: §E.13 · p. 176

E.14 Sensitivity to calibration dataset

• SLIM is largely insensitive to the calibration dataset, achieving comparable accuracy across different calibration datasets.The comparison evaluates SLIM-LoRA and SLIM-Quant perplexity using 4-bit quantization.Source: §E.14 · p. 177, Table E.16 · p. 177

E.15 Sparsity analysis

• SLIM-LoRA with SLIM-Quant maintains competitive accuracy up to 60% sparsity, whereas other methods degrade noticeably at lower sparsity levels.Increasing sparsity raises perplexity, revealing a compression–accuracy trade-off.Source: §E.15 · p. 177

E.16 Group quantization challenges

• Group quantization requires more complicated implementations and additional memory, with limited support across GPU architectures and software frameworks.The Marlin kernel is highly architecture-specific, while Triton and CUTLASS lack 4-bit group-quantization support.Source: §E.16 · p. 178
• Group quantization slows matrix multiplication because each group requires additional dequantization parameters, with measured slowdowns on LLaMA-2 and LLaMA-3.1 down-projection matrices.The evaluation uses an NVIDIA A100-40GB GPU with batch size 16.Source: §E.16 · p. 178, Table E.17 · p. 178