GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruning

1 INTRODUCTION

• LLM scale brings high computational and memory costs that complicate deployment on resource-constrained hardware and latency-sensitive applications.Compression seeks to reduce these costs while preserving general capabilities.Source: §1 · p. 1
• Prior work found non-uniform layer-wise ratios less effective under fine-grained N:M sparsity, leaving open whether the limitation is general or pattern-specific.N:M fixes the local sparsity ratio in every layer and module.Source: §1 · p. 2
• Group-level sparsity retains or prunes regular groups as wholes, letting each layer’s sparsity ratio vary under a global budget.Unlike N:M sparsity, it does not enforce the same local ratio in every layer.Source: §1 · p. 2
• GroupMask jointly generates layer group selectors with a lightweight hypernetwork, sparsity-budget regularization, and self-distillation while keeping pretrained weights frozen.The selectors use a Gumbel-Sigmoid parameterization with a straight-through estimator.Source: §1 · pp. 2–3
• At 50% sparsity with 1 × 256 groups on LLaMA-2-7B, adaptive allocation lowers WikiText-2 perplexity from 10.02 to 8.30 and raises average zero-shot accuracy from 0.455 to 0.496 versus a uniform ratio.GroupMask also outperforms evaluated baselines in WikiText-2 perplexity on LLaMA-2-7B and achieves the highest average zero-shot accuracy with Alpaca calibration across five LLaMA and Qwen models.Source: §1 · p. 2
• The paper evaluates GroupMask on five LLaMA and Qwen models using perplexity and six zero-shot benchmarks.Its analysis reports that LLaMA-2-7B retains 36.6% of attention parameters and 56.6% of MLP parameters.Source: §1 · p. 3

2.1 NOTATION AND GROUP SPARSITY PATTERN

• Each target weight matrix is partitioned into non-overlapping groups retained or pruned as a whole, so layer-adaptive allocation can vary by projection.Group-level selectors are expanded into full-resolution masks; using groups reduces binary decisions by a factor equal to group size, such as 256 for 1 × 256 groups.Source: §2.1 · p. 3

2.2 HYPERNETWORK-BASED MASK GENERATION

• A hypernetwork processes a fixed random vector sequence with a bidirectional GRU and layer-specific heads to generate selector logits jointly across target layers.Gumbel-Sigmoid relaxation with a straight-through estimator uses binary selectors in the forward pass and propagates gradients through a continuous relaxation in the backward pass.Source: §2.2 · p. 3

2.3 SPARSITY BUDGET REGULARIZATION

• The log-ratio regularizer penalizes deviation of the global sparsity ratio from its target while allowing individual layer ratios to differ.It is computed on discrete selectors; the uniform variant instead regularizes each layer’s ratio separately.Source: §2.3 · p. 3

2.4 SELF-DISTILLATION OPTIMIZATION FRAMEWORK

• Self-distillation compares dense teacher outputs with outputs from the masked student, optimizing the mask-generation hypernetwork while pretrained weights remain frozen.The distillation objective is equivalent, up to a teacher-only constant, to forward KL divergence from the teacher distribution to the student distribution.Source: §2.4 · p. 4

3.1 SETTINGS

• Experiments cover five LLaMA and Qwen models, WikiText-2 perplexity, and six zero-shot benchmarks, using 1 × 256 groups and 50% sparsity unless stated otherwise.Calibration uses WikiText or Alpaca; pretrained weights remain frozen, with no post-pruning fine-tuning.Source: §3.1 · p. 5, §3 · p. 5

3.2 LANGUAGE MODELING PERFORMANCE

• 8.30 versus 9.84: GroupMask has the lowest WikiText-2 perplexity among compared methods on LLaMA-2-7B at 50% sparsity.On LLaMA-2-13B, its perplexity is 7.19, second to DISP-LLM at 7.11; structured methods are compared at equal parameter sparsity, not equal inference cost.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 ZERO-SHOT DOWNSTREAM PERFORMANCE

• With Alpaca calibration, GroupMask has the highest average zero-shot accuracy among evaluated baselines on all five models; it also achieves 8.30 WikiText-2 perplexity on LLaMA-2-7B.On LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B, its averages exceed MoDeGPT by 3.0, 2.6, and 2.0 points; on Qwen3-8B and Qwen3-14B, they exceed DISP-LLM by 4.0 and 5.2 points.Source: §3.3 · pp. 5–6, §3.2 · p. 5, Table 1 · p. 5

3.4 LAYER-ADAPTIVE VERSUS UNIFORM ALLOCATION

• 10.02 to 8.30 perplexity and 0.455 to 0.496 average zero-shot accuracy: learned layer-adaptive allocation beats a uniform ratio under the same group pattern.The settings share group structure, calibration data, and training schedule; BoolQ accuracy decreases from 0.622 to 0.588.Source: §3.4 · p. 6, Table 3 · p. 6

3.5 ABLATION STUDY

• Removing the hypernetwork lowers average zero-shot accuracy from 0.496 to 0.403 with WikiText-2 calibration and from 0.523 to 0.387 with Alpaca calibration.Accuracy tends to decrease as group size grows, though not monotonically; Alpaca calibration yields higher averages than WikiText-2 for every tested group shape.Source: §3.5 · pp. 7–8, Table 4 · p. 7
• From 30% to 50% sparsity, LLaMA-2-7B perplexity rises from 7.32 to 9.37, while its average accuracy falls from 0.596 to 0.488.LLaMA-3-8B average accuracy also falls, from 0.580 to 0.481, over that sparsity range.Source: Figure 3 · p. 7, §3.5 · p. 8
• A preliminary group-sparse SpMM kernel reaches 1.09× speedup at 50% sparsity and 1.88× at 90% on an NVIDIA H100.These are kernel-level measurements; end-to-end acceleration requires further system support.Source: §3.5 · p. 8
• At 50% sparsity, LLaMA-2-7B preserves 36.6% of attention parameters and 56.6% of MLP parameters; deep layers 22–31 keep about 8% of Q and K groups.Q/K and Gate/Up preserved rates correlate at 0.994; projection type explains about half of preserved-rate variation across 224 projections.Source: §3.5 · p. 8, Figure 4 · p. 8

Pruning for LLMs

• Unstructured pruning removes individual weights but has irregular sparsity, while structured pruning removes larger components and yields dense models that run efficiently on standard hardware.The trade-off is flexible decisions versus coarser pruning decisions.Source: §4 · p. 8

Semi-structured sparsity

• GroupMask learns group-level masks for billion-parameter decoder-only LLMs in a task-agnostic, post-training setting, jointly generating layers’ masks with frozen weights.The approach differs from prior block pruning for BERT-scale encoders, which learns masks alongside weight updates during task-specific fine-tuning.Source: §4 · p. 9
• N:M sparsity fixes the number of retained weights in each local window, whereas group-level masks let each layer’s sparsity ratio vary above or below the global budget.PATCH’s tiles are at most 50% sparse, so at a 50% global budget its pattern reduces to uniform 2:4 sparsity.Source: §4 · p. 9

Layer-wise sparsity allocation

• Under N:M sparsity, non-uniform layer-wise ratios become less effective because critical weights may still be removed within each block; GroupMask tests adaptive allocation under group-level sparsity.For unstructured pruning, non-uniform ratios have improved over uniform ratios, whether derived from outlier statistics or learned.Source: §4 · p. 9

Learnable masks

• Learnable pruning methods optimize pruning decisions with gradient-based relaxations, and GroupMask uses a shared hypernetwork to generate group-level selectors for all layers.The selectors are trained through self-distillation while pretrained weights stay frozen.Source: §4 · p. 9

5 CONCLUSION

• With group-level sparsity on LLaMA-2-7B, adaptive allocation lowers WikiText-2 perplexity from 10.02 to 8.30 relative to a uniform ratio, suggesting the earlier N:M limitation is pattern-specific.With Alpaca calibration, GroupMask has the highest average zero-shot accuracy among evaluated baselines on five LLaMA and Qwen models.Source: §5 · p. 9

AI USE STATEMENT

• The authors report using generative AI tools for research discussions, software, analysis, and manuscript preparation, and say they reviewed and verified the resulting work.They state that reported experimental results came from their implementations and experiments, not AI generation.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The paper identifies its methodology, training details, calibration settings, additional results, mask export procedures, and kernel evaluations as reproducibility materials.The implementation is available in the repository linked in the abstract.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The work uses publicly available pretrained models and benchmark datasets and involves neither human subjects nor personal-data collection.Source: ETHICS STATEMENT · p. 10

• The dense pretrained model serves as teacher and the masked model as student; only hypernetwork parameters are updated during mask learning.Teacher logits use masks disabled and are detached, while student logits use group-sparse masks; pretrained weights stay frozen.Source: A IMPLEMENTATION DETAILS · p. 14
• WikiText and Alpaca are the calibration datasets, with a fixed seed of 42 unless stated otherwise and a maximum calibration sequence length of 2048.The two settings are denoted GroupMask-Wiki and GroupMask-Alpaca.Source: A IMPLEMENTATION DETAILS · p. 14
• The default experiments use 1 × 256 groups and 50% sparsity, while granularity and sparsity analyses vary group shapes and target sparsity.The evaluated group shapes include 16 × 16, 1 × 128, 1 × 256, 32 × 32, 64 × 64, and 128 × 128; sparsity settings include 30%, 40%, and 50%.Source: A IMPLEMENTATION DETAILS · p. 14
• Language-model quality is measured by WikiText-2 perplexity, and zero-shot accuracy is reported on six benchmarks and averaged across tasks.Zero-shot evaluation is conducted without task-specific fine-tuning.Source: A IMPLEMENTATION DETAILS · p. 14

• For LLaMA-2-7B and LLaMA-3-8B, average zero-shot accuracy decreases monotonically as sparsity increases from 30% to 50%.Source: Table 6 · p. 15, B ADDITIONAL SPARSITY SENSITIVITY… · p. 15
• The zero-shot sweep reports six benchmarks and their average score, while the accompanying perplexity sweep uses 32 × 32 groups.The zero-shot results described for these sparsity levels use 32 × 32 groups and WikiText-2 calibration.Source: B ADDITIONAL SPARSITY SENSITIVITY… · p. 14, Table 6 · p. 15, Table 7 · p. 15
• For LLaMA-2-7B and LLaMA-2-13B, WikiText-2 perplexity increases monotonically as sparsity rises from 30% to 50%.The table reports perplexity for the 32 × 32 group setting, with lower values indicating better language-model performance.Source: Table 7 · p. 15, B ADDITIONAL SPARSITY SENSITIVITY… · p. 15

C SQUARE GROUP GRANULARITY

• Table 8 evaluates 32 × 32, 64 × 64, and 128 × 128 groups across multiple model families using WikiText-2 and Alpaca calibration.Source: C SQUARE GROUP GRANULARITY · p. 15
• 32 × 32 gives the best square-group performance for LLaMA-2-13B under Alpaca calibration, with an average score of 0.562.Source: C SQUARE GROUP GRANULARITY · p. 15
• 32 × 32 gives LLaMA-3-8B the best square-group performance, with average scores of 0.481 using WikiText-2 and 0.488 using Alpaca calibration.Source: C SQUARE GROUP GRANULARITY · pp. 15–16
• 32 × 32 Alpaca achieves the highest average score among square-group settings on Qwen3-8B and reaches 0.561 on Qwen3-14B.Source: C SQUARE GROUP GRANULARITY · p. 16
• 128 × 128 underperforms 32 × 32 in every Table 8 setting, while 1 × 128 and 1 × 256 obtain higher averages than 32 × 32 and larger square groups in Table 4.A 128 × 128 group covers 16 times as many weights as a 32 × 32 group.Source: C SQUARE GROUP GRANULARITY · p. 16

D ADDITIONAL ABLATION AND EFFICIENCY ANALYSIS

• Figure 5a shows sparsity-regularization loss decreasing faster with the hypernetwork than without it.This complements the task-loss comparison in Figure 3a.Source: Figure 5 · p. 16, D ADDITIONAL ABLATION AND EFFICIENCY… · p. 16
• At 50% sparsity, kernel speedup varies only slightly across the evaluated group sizes.Source: Figure 5 · p. 16, D ADDITIONAL ABLATION AND EFFICIENCY… · p. 16
• Kernel speedup increases with sparsity under group size 64.Source: Figure 5 · p. 16, D ADDITIONAL ABLATION AND EFFICIENCY… · p. 16

E BASELINE AND EVALUATION DETAILS

• Published baseline numbers are reported when available under the same setting.Source: E BASELINE AND EVALUATION DETAILS · p. 16
• Entries are left blank for models or settings absent from prior work or not directly applicable.Source: E BASELINE AND EVALUATION DETAILS · p. 16
• All zero-shot evaluations use lm-evaluation-harness with the official task metrics.An asterisk marks a result taken from the corresponding paper or following its reported setting.Source: E BASELINE AND EVALUATION DETAILS · p. 16

F MASK EXPORT AND PRESERVED RATE COMPUTATION

• After mask learning, continuous mask scores are thresholded into binary group selectors using the evaluation thresholding rule, then applied to the original pretrained weights.Source: F MASK EXPORT AND PRESERVED RATE… · p. 16
• For each projection matrix, preserved rate is the number of active weights divided by the total number of weights after applying the binary masks.Source: F MASK EXPORT AND PRESERVED RATE… · p. 16
• For QKV projections, the reported preserved rate averages the query, key, and value projection-matrix rates.Source: F MASK EXPORT AND PRESERVED RATE… · p. 16