PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

Tianxiang Xia, Kai Lion, Niao He, Michael Muehlebach, Melanie Zeilinger, Hao Ma

arXiv:2610.31897v1cs.LGcs.AI

TL;DR

Large-model edge inference faces memory and latency constraints, while cloud connectivity, delays, and privacy can limit remote execution. S-SALAAD learns structured sparse-plus-low-rank representations and pairs them with budget-aware compression and dedicated kernels; across Llama pretraining and Qwen distillation it advances the empirical perplexity–memory–latency Pareto frontier.

  • Problem

    Large models face memory and inference-latency constraints on edge devices, while connectivity, delay, and privacy requirements can make cloud inference impractical.

    Source: §1 · p. 1

  • Method

    S-SALAAD jointly learns model weights and structured sparse-plus-low-rank components during pretraining, then supports budget-aware compression and dedicated CPU and GPU kernels.

    Source: §1 · p. 2

  • Results

    Across conventional Llama pretraining and cross-architecture Qwen distillation, S-SALAAD advances the empirical perplexity–memory–latency Pareto frontier; at 500M, throughput rises from 29.35 ktoken s−1 to 58.47 ktoken s−1 relative to dense inference.

    Source: §5.1 · pp. 7–8

  • Takeaways & Limitations

    A single S-SALAAD checkpoint supports elastic deployment across parameter budgets, and the analysis links tolerable block granularity to model scale.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unreliable network connectivity, communication delays, and privacy requirements can make such reliance impractical, motivating local execution on edge devices. Local deployment requires models to fit within limited memory and execute efficiently with the available computational resources. Recent work has shown that learning sparse-plus-low-rank representations enables elastic compression of a single checkpoint to multiple memory budgets. However, fewer parameters do not necessarily translate into faster inference. Indexing overhead and irregular memory access in unstructured sparse components can offset computational savings. To address this critical deployment barrier, we introduce S-SALAAD, a unified framework that jointly learns the model weights and their low-rank and structured sparse components during training. The framework supports a family of hardware-compatible structured sparsity patterns. It couples this structure learning with budget-aware compression and dedicated CPU and GPU kernels that exploit the learned structure for acceleration, enabling efficient inference from a single checkpoint across memory budgets. Experiments on Llama2 and Qwen3 models, covering both pretraining and distillation, show that S-SALAAD advances the empirical Pareto frontier in perplexity, memory, and inference latency across multiple hardware platforms relative to the evaluated baselines. These results show that structured learning can translate elastic compressibility into reduced memory requirements and lower inference latency, enabling practical deployment of large models on edge devices.

1 INTRODUCTION

Large models strain edge-device memory and latency budgets, while connectivity, delay, and privacy constraints can make cloud inference impractical. S-SALAAD learns structured sparse-plus-low-rank representations during pretraining and supports elastic compression and efficient inference.

1 INTRODUCTION

  • Edge deployment must meet both memory and inference-latency constraints despite unreliable connectivity, communication delays, and privacy requirements.Latency is especially consequential in feedback control, where added loop delay can degrade performance or destabilize a system.Source: §1 · p. 1
  • Unstructured sparse-plus-low-rank representations can reduce static memory yet increase inference latency because irregular memory access offsets computational savings.Post-hoc conversion to structured patterns can sharply degrade performance, motivating structure induction during pretraining.Source: §1 · p. 2
  • S-SALAAD jointly learns weights and low-rank and structured sparse components during pretraining, supporting block, N:M, and row-/column-wise patterns without changing architecture or optimizer rules.It also uses budget-aware refinement for zero-shot compression and dedicated CPU and GPU kernels.Source: §1 · p. 2
  • Across Llama2 pretraining and Nemotron-3-Super-120B-to-Qwen3-1.7B distillation, S-SALAAD advances the empirical perplexity–memory–latency Pareto frontier; its analysis links tolerable block granularity to model scale.The analysis identifies block sparsity as a practical choice for flexible compression and efficient CPU and GPU execution.Source: §1 · p. 2

2 RELATED WORK

Large-language-model compression includes pruning, low-rank factorization, and quantization, as well as post-training and compression-aware training approaches. S-SALAAD learns structured sparsity during training while preserving the original architecture and supporting elastic budgets from one checkpoint.

2 RELATED WORK

  • Compression approaches include pruning, low-rank factorization, and quantization, with post-training methods and compression-aware training offering different routes to smaller models.Post-training methods include iterative ADMM approaches, while training-time methods induce compressibility during pretraining.Source: §2 · pp. 2–3
  • Unstructured sparse components in SLTrain and SALAAD do not map efficiently to accelerator primitives, while LOST fixes model size and changes the architecture.ELSA and 3BASiL are data-hungry and take hours of GPU time to compress a single model.Source: §2 · p. 3
  • S-SALAAD preserves the original architecture and learns structured sparsity, allowing one checkpoint to support multiple parameter budgets.This contrasts with LOST's preset rank and sparse capacity, which fix model size during training.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation decomposes each target projection into low-rank and sparse components optimized under the task loss. A group-based regularizer determines the sparse structure, with several hardware-friendly patterns as instantiations.

3 PROBLEM FORMULATION

  • SALАAD decomposes each target projection matrix X into low-rank L and element-wise sparse S components under a task-specific loss.The optimization imposes X = L + S and penalizes the nuclear and element-wise norms.Source: §3 · p. 3
  • The structured formulation replaces the element-wise sparse penalty with a regularizer over non-overlapping groups, jointly optimizing X, L, and S under the task loss.The group partition and assigned norm determine the sparsity structure; singleton groups with absolute-value norms recover element-wise sparsity.Source: §3 · p. 3
  • Block sparsity treats each p × q block as a group and penalizes its Frobenius norm, removing blocks as units.The described instantiations are not exhaustive; the framework can support other structures as hardware and deployment needs evolve.Source: §3 · p. 3
  • Row- and column-wise sparsity group entire rows or columns, while N:M sparsity partitions each row into M-entry groups targeting N retained nonzeros per group.Setting the block dimensions to (1, m) or (n, 1) gives row- or column-wise groups, respectively.Source: §3 · p. 4

4 METHODOLOGY

S-SALAAD uses ADMM to learn structured sparse-plus-low-rank components during training, then adds efficient low-rank updates, budget-aware compression, and block-sparse inference kernels. Its implementation uses warm-started iterative SVD, energy-guided allocation and refinement, and dedicated CPU and GPU kernels.

4.1 STRUCTURED SLR LEARNING

  • ADMM alternates updates to model weights, low-rank and structured sparse components, and a dual variable to jointly learn the decomposition.The model update uses gradient-based optimization and permits existing optimizers without modification.Source: §4.1 · pp. 4–5
  • The low-rank proximal step applies singular-value soft thresholding, while the structured sparse proximal problem decomposes into independent group-wise subproblems.The group update uses group soft thresholding for block and row-/column-wise sparsity.Source: §4.1 · pp. 4–5

4.2 END-TO-END IMPLEMENTATION

  • Warm-started I-SVD uses one orthogonal-iteration step from a cached right-singular basis to approximate singular triplets and track effective rank.The paper reports that this step closely tracks exact-SVD effective-rank trajectories while preserving compression quality.Source: §4.2 · p. 5
  • 29%: I-SVD reduces mean optimizer-step time from 890 ms with exact SVD to 632 ms, leaving 4% overhead versus the 609 ms vanilla baseline.Figure 1 gives the per-step training-time breakdown for Llama-1B on 8 H100 GPUs.Source: §4.2 · pp. 5–6, Figure 1 · p. 5
  • EGAR allocates a target parameter budget across projections and low-rank and sparse capacities, then refines both components to improve dense-weight reconstruction without calibration data or further training.Stage I uses per-projection thresholds and returns retained ranks and block counts within the total budget.Source: §4.2 · p. 6
  • Stage II alternates low-rank and sparse residual projections within the allocated capacities, keeping reconstruction error non-increasing while preserving the parameter budget.The low-rank subspace and block support remain free to change within those capacities.Source: §4.2 · p. 6
  • Dedicated block-sparse CPU and GPU kernels add the sparse contribution directly to the low-rank output and exploit contiguous retained blocks.The GPU implementation uses TileLang with Tensor Cores, while the CPU kernel is vectorized with AVX2.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Experiments evaluate S-SALAAD on Llama pretraining and Qwen distillation across compression budgets, hardware, and benchmark measures. Results report Pareto improvements and examine downstream accuracy, sparsity patterns, and block-size trade-offs.

5 EXPERIMENTS

  • The experiments pretrain Llama2 1B and 350M models on C4 and distill Nemotron-3-Super-120B into Qwen3-1.7B on FineWeb-Edu.Evaluation includes perplexity, zero-shot downstream accuracy, peak memory, and prefill latency, with inference profiled on two GPUs and a CPU host configuration.Source: §5 · p. 7

5.1 MAIN RESULTS

  • At 500M parameters, block-64 × 64 S-SALAAD reduces peak GPU memory from 4.08 GB to 2.35 GB and increases throughput from 29.35 ktoken s−1 to 58.47 ktoken s−1 versus dense inference.On a Quadro RTX 6000, the reported Pareto comparison also finds S-SALAAD superior to Wanda, SparseGPT, SliceGPT, and SVD-LLM; Qwen distillation results extend the gains across model families and training paradigms.Source: §5.1 · pp. 7–8, Figure 4 · p. 8
  • S-SALAAD-pretrained L-only checkpoints improve SliceGPT and SVD-LLM trade-offs relative to their vanilla-pretrained counterparts.The reported comparison describes synergy between learned compressibility and post-training compression.Source: §5.1 · p. 8
  • Zero-shot downstream accuracy degrades gracefully as parameter budgets tighten across Llama and Qwen compressed variants.The evaluation compares vanilla and S-SALAAD models across budgets.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 ABLATION STUDIES

  • Block and row-/column-wise sparsity maintain competitive perplexity across wider parameter-budget ranges than 2:4 sparsity on Llama-350M.Although 2:4 has lower perplexity at the largest budgets, its fixed sparse capacity restricts elastic compression as parameter count decreases.Source: §5.2 · p. 8, Figure 5 · p. 9
  • At tight budgets on Llama-1B, 8×8 and 16×16 blocks achieve perplexities of 17.07 and 17.03, respectively, compared with 17.86 for L-only at 500M parameters.Finer blocks preserve quality, while their additional inference overhead limits deployment gains; coarser blocks can be more suitable at larger model scales.Source: §5.2 · p. 9, Figure 6 · p. 9

6 CONCLUSION

S-SALAAD learns structured sparse-and-low-rank representations during training for elastic edge deployment. Across Llama pretraining and Qwen distillation, it advances the empirical perplexity-memory-latency Pareto frontier relative to evaluated baselines.

6 CONCLUSION

  • Zero-shot budget-aware compression and dedicated block-sparse CPU and GPU kernels support memory savings and efficient inference across budgets from one checkpoint.The conclusion also links tolerable block granularity to model scale as empirical guidance for extending the framework.Source: §6 · p. 9

USE OF LARGE LANGUAGE MODELS

The authors report using generative AI tools for feedback, implementation, translation, dataset cleanup, plotting, figure editing, and manuscript readability. They state that the resulting work was reviewed and verified by authors.

USE OF LARGE LANGUAGE MODELS

  • Authors used generative AI tools for methodological and experimental feedback, implementation, translation, dataset cleanup, plotting and figure editing, and manuscript readability.They state AI tools did not design the research methodology or experiments, develop theoretical models, formulate mathematical claims, or interpret results; two authors verified implementations, processed data, plotting code, and figures.Source: USE OF LARGE LANGUAGE MODELS · p. 10

REPRODUCIBILITY STATEMENT

The appendices document the evaluation setup and examine latency costs, post-training block constraints, L-only compression, controller behavior, iterative SVD, and EGAR. The reported comparisons show where these methods trade off quality, memory, and inference efficiency.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement lists datasets, tokenization, configurations, hyperparameters, parameter counts, baselines, evaluation protocols, and makes checkpoints, teacher logits, and code available.Evaluation protocols include hardware, precision, input shapes, timing, and memory measurement.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 MEMORY SAVINGS DO NOT GUARANTEE LOWER INFERENCE LATENCY

  • Unstructured SLR cuts static weight memory from 23.07 MB to 10.49 MB, but throughput falls to 2.68% of dense, implying 37.3× latency for this workload.The benchmark uses a synthetic Llama-1B MLP down projection with 15% rank ratio and 5% density.Source: Figure 7 · p. 14, §A.1 · p. 14

A.2 QUALITY DEGRADATION FROM IMPOSING BLOCK STRUCTURE AFTER PRETRAINING

  • Post-hoc block compression raises perplexity as blocks become coarser, with steeper increases at lower budgets; the authors attribute this to reduced pruning flexibility.The comparison uses the same unstructured SALAAD Llama-1B checkpoint across three parameter budgets; the first table row is the element-wise reference.Source: Table 2 · p. 14, §A.2 · p. 14

B L-ONLY S-SALAAD

  • L-only S-SALAAD maintains low perplexity across tested budgets and offers a better quality–elastic-compressibility trade-off than direct subgradient regularization baselines.The comparison covers λ = 10^-6 and 10^-5; the latter keeps perplexity nearly constant across budgets at the cost of higher perplexity throughout.Source: B L-ONLY S-SALAAD · p. 15, Figure 8 · p. 15

C.1 ADAPTIVE I-CONTROLLER

  • The I-controller tracks target effective-rank ratio and group sparsity by adjusting α and β, using group-level sparsity to match the granularity of parameter removal.Both statistics also guide EGAR Stage I, which recomputes them after thresholding to adapt the next threshold update.Source: §C.1 · pp. 15–16

C.2 FIDELITY OF ITERATIVE SVD

  • For three representative Llama-350M down projections, exact and iterative SVD trajectories are indistinguishable in the converged regime.The appendix also compares their post-compression perplexity at matched budgets.Source: Figure 10 · p. 16, Figure 9 · p. 16

C.3 COMPRESSION BY ENERGY-GUIDED ALTERNATING REFINEMENT

  • At both 400M and 500M parameters, EGAR achieves lower perplexity than HPA for every evaluated κ from 0.5 to 0.9.Alternating refinement monotonically reduces normalized reconstruction error and converges in fewer than 10 iterations, taking a few minutes.Source: Figure 12 · p. 17, §C.3 · p. 17

C.4 KERNEL MICRO-BENCHMARK

The kernel micro-benchmark tests structured sparse inference, while the appendices specify the hardware proxy and model data and configurations. The results show that measured speed depends on both block size and the kernel implementation.

C.4 KERNEL MICRO-BENCHMARK

  • 1.49×, 1.87×, and 2.33× dense throughput for 16×16, 32×32, and 64×64 blocks; PyTorch BSR remains slower than dense at all three sizes.In this setting, 64×64 blocks reach 1.57× the throughput of 16×16 blocks.Source: §C.4 · pp. 17–18, Figure 14 · p. 18

D.1 RELEVANCE TO EDGE DEPLOYMENT

  • The edge-deployment evaluation uses Quadro RTX 6000 and RTX 2080 Ti GPUs and CPU-only inference on 4× AMD EPYC 7662 CPUs; energy consumption and thermal management are out of scope.These workstation/server platforms are used as proxies for edge inference based on memory capacity and available acceleration.Source: §D.1 · p. 18

D.2 DATASETS AND MODEL CONFIGURATIONS

  • Llama 2 pretraining uses C4 with t5-base tokenization, while Qwen3-1.7B distillation uses FineWeb-Edu and the teacher’s Nemotron tokenizer for a shared token space.Llama-1B supports the main block-sparse experiments; Llama-350M is used for sparsity-pattern comparisons.Source: §D.2 · pp. 18–19, Table 3 · p. 19

D.3 TRAINING CONFIGURATIONS

The appendix documents training, evaluation, and profiling settings, then examines convergence and learned sparsity patterns. It reports paired head-level structure across attention projections and sparsity granularities.

D.3 TRAINING CONFIGURATIONS

  • Controller step sizes can generally be reused across model scales within a sparsity variant, with ρ adjusted by the empirical scaling rule.The target effective-rank ratio is 0.15 and target sparsity 0.95, except 2:4 sparsity, which uses 0.03 and 0.5.Source: §D.3 · pp. 19–20

D.4 EVALUATION AND BASELINES

  • Evaluation covers held-out-corpus perplexity and zero-shot accuracy, with vanilla training, SALAAD, L-only, and compression baselines evaluated without further fine-tuning.Wanda and SparseGPT use dense-model latency and throughput because the hardware lacks native 2:4 acceleration.Source: §D.4 · p. 20

D.5 PROFILING ENVIRONMENT AND PROTOCOL

  • Profiling uses fixed input shapes, warmups, and timed passes on GPUs and CPUs, reporting median latency and excluding data loading, input transfer, compilation, and loss evaluation.The timed forward measures prefill rather than autoregressive decoding or complete request latency.Source: §D.5 · p. 21, Table 8 · p. 21

E.1 TRAINING LOSS

  • Across the evaluated settings, structured sparsity preserves stable convergence with modestly higher task loss than vanilla training, while Llama trajectories remain comparable to unstructured SALAAD.The Qwen3-1.7B distillation comparison uses the same objective, data, optimizer, and schedule for both runs.Source: §E.1 · p. 22
  • Structured variants show stable training loss, with a modest increase relative to vanilla training and Llama trajectories comparable to unstructured SALAAD.The loss curves are moving averages over approximately 1,000 training steps and end at the last logged step.Source: §E.1 · p. 22, Figure 15 · p. 22

E.2 LEARNED SPARSITY PATTERNS

  • Learned components display the intended 2:4 and block patterns, while block-sparse components also show recurring row- or column-aligned clustering.The 2:4 components have approximately 50% density, but occupancy varies across four-entry groups; projection in EGAR Stage II enforces at most two nonzeros per group.Source: §E.2 · p. 22, Figure 16 · p. 23, Figure 17 · p. 24

E.3 PAIRED SPARSITY PATTERNS

  • Sparse components show paired head-level patterns between Q and K and between V and O, and dominant head locations are broadly consistent across examined granularities.The reported consistency concerns overlap or distribution across heads, not identical nonzero heads or identical block supports.Source: §E.3 · pp. 25–27

E.4 EFFECTIVE-RANK AND SPARSITY RATIOS

During training, sampled projections progressively approach their prescribed effective-rank and sparsity targets. This pattern holds across the evaluated sparsity patterns, model sizes, and architectures, though convergence behavior differs by layer and projection type.

E.4 EFFECTIVE-RANK AND SPARSITY RATIOS

  • The trajectories cover block-16×16, row-/column-wise, and 2:4 variants on Llama-350M; four block sizes on Llama-1B; and block-64×64 Qwen3-1.7B distillation.Figures 20–22 track effective-rank ratio and sparsity in sampled projections.Source: §E.4 · p. 27, Figure 20 · p. 27, Figure 21 · p. 28
  • Across evaluated sparsity patterns, model sizes, and architectures, components progressively approach their prescribed structure, with convergence varying across layers and projection types.The targets are effective-rank ratio 0.15 and sparsity 0.95; the 2:4 pattern instead uses fixed 50% sparsity.Source: §E.4 · p. 27

F.1 PARETO FRONTIERS ON OTHER HARDWARE

Across additional GPU and CPU platforms, block-64×64 S-SALAAD reduces memory use and increases throughput in the reported Llama-1B and Qwen3-1.7B settings. The appendix also examines fixed-memory Qwen inference and how block granularity changes deployment trade-offs.

F.1 PARETO FRONTIERS ON OTHER HARDWARE

  • 42.4% lower peak GPU memory and 2.25× throughput for Llama-1B, and 31.5% lower memory and 1.80× throughput for Qwen3-1.7B, are reported for block-64×64 on RTX 2080 Ti.On 4× AMD EPYC 7662 CPUs, the corresponding Llama-1B gains are 61.5% lower peak RAM and 2.78× throughput; Qwen3-1.7B gains are 50.6% and 2.20×.Source: §F.1 · p. 29, Figure 23 · p. 29, Figure 24 · p. 30

F.2 QWEN INFERENCE UNDER A FIXED MEMORY CAP

  • At a 1B parameter budget under a 3.4 GB peak-memory cap, Qwen3-1.7B S-SALAAD achieves 2.48× dense-inference throughput, while perplexity increases from 12.71 to 14.06.The batch size is increased for each model within the memory cap.Source: §F.2 · p. 30, Figure 25 · p. 30

G BLOCK GRANULARITY AND THE PARETO FRONTIER

  • On Llama-1B, the coarsest blocks retain competitive quality and closely track L-only deployment trade-offs on both GPUs, while on CPU the same granularity offers operating points that improve on L-only.Source: G BLOCK GRANULARITY AND THE PARETO… · p. 31

H NUMERICAL RESULTS

  • Tables 11–14 list operating points for the plotted deployment comparisons, including additional Figure 26 points not already reported in Tables 11–13.The tables specify the hardware, precision, input shapes, preferred metric directions, and settings represented.Source: H NUMERICAL RESULTS · p. 31, Table 11 · p. 32, Table 12 · p. 33

Related papers

Loading 2610.31897v1…