LaCo: Large Language Model Pruning via Layer Collapse

• Existing compression approaches face hardware requirements, costly retraining, performance loss, or altered model structure that reduces portability.Source: §1 · p. 1
• LaCo collapses rear model layers into a prior layer to reduce model size while preserving the model’s output representation.Source: §1 · pp. 1–2
• 30%-50% of model layers can be removed without training while maintaining performance, and LaCo outperforms state-of-the-art structured pruning methods at equivalent pruning ratios.Source: §1 · p. 2
• LaCo preserves internal LLM structure, including intermediate dimensions, so pruned models can integrate into existing applications without implementation changes.Source: §1 · p. 2
• Post-training experiments, ablations, and evaluations across pruning ratios examine parameter inheritance and LaCo’s performance behavior.Source: §1 · p. 2

2.1 Reserving-Differences-while-Seeking-Common Layer Merge

• RDSC Layer Merge adds parameter differences from following layers to a retained layer, then discards those consecutive layers while applying the process independently to self-attention and MLP parameters.Source: §2.1 · p. 2

2.2 Layer Collapse

• Layer Collapse dynamically merges adjacent layers from the topmost layer downward and uses few-shot calibration samples to keep pruned and original output representations similar.Source: §2.2 · p. 2

(2) Pruning (line 1-17)

• Each pruning iteration merges up to C−1 subsequent layers into layer l, computes average cosine similarity between pruned and original hidden-state representations, and accepts the merge when s exceeds T.Source: (2) Pruning (line 1-17) · p. 3
• The procedure continues until the pointer falls below L, outputting the pruned LLM after updating the model and pointer differently for accepted and rejected merges.Source: (2) Pruning (line 1-17) · p. 3

2.3 Complexity Analysis

• Worst-case time complexity is O(H × ||D||), with Llama2-13B requiring at most 400 inference sentences for 40 layers and 10 calibration sentences.Source: §2.3 · p. 3

3.1 Models

• The evaluation covers Llama2-7B/13B and bilingual Baichuan2-7B/13B using OpenCompass across reasoning, language, knowledge, examination, and understanding tasks.Evaluations are zero-shot or few-shot without additional training, using PPL and GEN modes.Source: §3.1 · p. 3, §3.2 · pp. 3–4

3.3 Baselines

• LLM-Pruner and SliceGPT serve as structured-pruning baselines with equivalent or slightly smaller pruning ratios, while settings target ratios near 30%.Calibration uses randomly selected Wikipedia sentences, and experiments run on eight Nvidia A100 80GB GPUs.Source: §3.3 · p. 4, §3.4 · p. 4

3.5 Main Results

• LaCo achieves the best results on most benchmarks and surpasses baselines in overall averages despite slightly higher pruning ratios.It leads in four of five categories, remains comparable in reasoning, and exceeds baselines in average performance relative to Dense∗.Source: §3.5 · p. 4, Table 1 · p. 5
• LaCo maintains stable generation performance on CHID, XSUM, and WSCG, whereas baseline-pruned models can produce zero scores or repetitive outputs.An XSum example shows LaCo producing normal-looking sentences while baseline outputs are nonsensical repetitions.Source: §3.5 · pp. 4–5
• LaCo-pruned models preserve performance across Llama2 and Baichuan2 models, including larger-scale Llama2-70B evaluations.The authors report that LaCo outperforms the baseline on Llama2-70B and maintains performance above 70% on most benchmarks without crashes.Source: §3.5 · pp. 4–5

3.6 Comparison of Perplexity

• At 27% sparsity on Llama2-7B, LaCo achieves lower PPL than other evaluated pruners, based on evaluation using 500 sentences selected from Wikipedia.Source: §3.6 · pp. 5–6

3.7 Pruning Time

• LaCo pruning is more efficient than the baselines at 27% sparsity on Llama2-7B when measuring the main pruning process on one A100 GPU.The timing comparison excludes model and data loading and model storage.Source: §3.7 · p. 6, Table 4 · p. 6

3.8 Memory Usage and Inference Speed

• For Llama2-13B, LaCo-pruned models consume less memory and achieve faster inference than the compared pruned models.Unlike some baselines, LaCo does not reduce inference speed relative to the dense model.Source: §3.8 · p. 6, Table 5 · p. 6

4.1.1 Post-training

• Pruned Llama2-7B and Baichuan2-7B converge rapidly after post-training, reaching losses around 1.6 and 2.0 versus reported 1.75 and 1.9, at one-thousandth scratch-training cost.Post-training uses four Nvidia A100 80GB GPUs, taking approximately 28 hours for Llama2-7B and 35 hours for Baichuan2-7B.Source: §4.1.1 · p. 6
• Post-training significantly improves pruned Llama2-7B, whereas Baichuan2-7B shows mixed benchmark changes and a slight overall decline, attributed to data-distribution differences.The consistent Llama2-7B improvement supports effective parameter inheritance; LaCo also outperforms LLM-Pruner after post-training while using fewer resources.Source: §4.1.1 · pp. 6–7

4.1.2 Re-pruning

• With 55% of original parameters retained after re-pruning, Llama2-7B preserves about 70% of the original 7B model performance.The authors qualify this result because their training-data quality and scale are limited.Source: §4.1.2 · p. 7

4.2 Layer-wise Similarity

• Adjacent layers have small parameter changes and highly similar representations, with representation similarity typically near 1 from layers 3 to 28.Corresponding-matrix L2 values are generally no more than 200 for large matrices.Source: §4.2 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
• Merging every four consecutive layers preserves representations, with the lowest 4096-dimensional output cosine similarity above 0.996.This experiment supports RDSC Layer Merge as a mechanism for replacing multiple layers with one.Source: §4.2 · pp. 7–8

4.3 Varying Pruning Ratio

• Performance declines as pruning increases, but remains relatively stable from roughly 10–15% to 25% pruning and retains approximately 70% performance near 50% pruning.The evaluation covers Llama2-7B and Llama2-13B across approximately 10%, 25%, and 50% pruning ratios.Source: §4.3 · p. 8, Table 7 · p. 8

Model Quantization and Knowledge Distillation

• Quantization can require special hardware and affect performance, while knowledge distillation requires costly, task-specific retraining of a smaller model.Examples include SmoothQuant, GPTQ, QLoRA, OmniQuant, and several distillation methods.Source: §5 · pp. 8–9

Model Pruning

• Pruning methods remove parameters or structures, with SparseGPT addressing sparse regression, SpQR isolating outliers, and LLM-Pruner removing non-critical structures using gradient information.Structured pruning primarily removes parts of model modules.Source: §5 · p. 8

LaCo Compared with Prior Work

• Unlike layer-skipping or dropping methods, LaCo collapses layers, preserves the intrinsic model structure, and requires no training to select layers.The paper presents LaCo as a smaller, faster, more memory-efficient structured pruner with strong performance.Source: §5 · p. 9

6 Conclusion

• LaCo merges rear model layers into preceding layers for rapid model-size reduction without special hardware support while preserving the model’s intrinsic structure.The method also exposes potential parameter redundancy in existing LLMs.Source: §6 · p. 9
• Experiments show that LaCo significantly outperforms current state-of-the-art structured pruning methods and effectively inherits original model parameters after post-training.The study also includes ablations, layer-wise similarity analysis, and evaluations at different pruning ratios.Source: §6 · p. 9

Limitations

• LaCo cannot directly control the pruning ratio and instead requires tuning hyperparameters such as representation-similarity threshold T.The authors leave more systematic mappings from hyperparameters to target ratios for future work.Source: Limitations · p. 9
• LaCo lacks a complete theoretical proof, and the authors note that better layer-merging methods may exist despite the current approach’s effectiveness.These limitations are shared in part with the cited structured-pruning baselines.Source: Limitations · p. 9

B.1 Impact of Number of Layers Merged per Operation C

• For 27% compression, merging 3–6 layers maintains relatively good performance, while overly large C can stop pruning early; the authors recommend C between 4 and 7.Very small C also degrades performance.Source: §B.1 · p. 12, Table 10 · p. 12

B.2 Impact of Minimum Interval I

• At 27% sparsity, I values from 1 to 3 perform well, whereas I = 4 shifts merges toward earlier layers and reduces performance; the authors choose I = 2.The paper associates earlier-layer modification with performance drops.Source: §B.2 · p. 12, Table 11 · p. 12

B.3 Impact of Dataset D

• Randomly selected calibration sentences produce results on nearly the same scale, indicating little impact from dataset choice at the tested 27% compression ratio.A separate high-quality-corpus experiment likewise reports little calibration-dataset impact at the same compression rate.Source: §B.3 · p. 12, Table 12 · p. 12

B.4 Impact of Threshold T

• Lowering threshold T increases compression but worsens performance, establishing the expected compression–quality trade-off.The ablation tests T values of 0.85, 0.65, 0.45, and 0.25 on Llama2-7B.Source: §B.4 · pp. 12–13, Table 13 · p. 13

B.5 Different Similarity Metrics

• KL divergence and CKA remain nearly constant because adjacent output distributions differ little and high-dimensional distances lack discrimination, so LaCo uses cosine similarity.Cosine similarity is retained as the more discriminative metric.Source: §B.5 · p. 13, Table 14 · p. 13

B.6 Drop or Merge

• At equal compression, merging outperforms simply dropping layers, while iterative LaCo merging outperforms the tested rule-based grouping strategy.Even rule-based merging can outperform baselines across multiple datasets.Source: §B.6 · p. 13, §B.7 · p. 13, Table 15 · p. 13

B.8 Effect of Calibration Dataset from Different Sources

• Different high-quality calibration datasets have little impact on LaCo performance at the same compression rate.The comparison prunes Llama2-7B at 27% compression using 10 BookCorpus sentences and repeats the experiments three times.Source: §B.8 · pp. 13–14

B.9 Effect of Calibration Dataset Size

• Calibration-set size has little impact on LaCo results when pruning Llama2-7B at 27% compression.The tested sizes are 5, 10, 50, 100, and 200 Wikipedia samples.Source: §B.9 · p. 14
• LaCo uses a small calibration dataset of 10 samples to balance pruning effect and pruning speed.Source: §B.9 · p. 14

• Post-training uses LLaMA-Factory with DeepSpeed ZeRO-2 and a sequence length of 4096.Source: C Post-Training Implementation Details · p. 14
• The setup uses Adam at 2e-4, total batch size 32, four-step gradient accumulation, cosine scheduling, 0.1 weight decay, and maximum gradient normalization 1.0.The Adam coefficients are β1 = 0.9 and β2 = 0.95, with batch size 8 per GPU.Source: C Post-Training Implementation Details · p. 14

D.1 Results on Llama2-70B

• At 30% compression on Llama2-70B, LaCo performs better than the baselines across the reported benchmark average.The evaluation covers CMNLI, HeSw, PIQA, and BoolQ.Source: §D.1 · p. 14, Table 19 · p. 14

D.3 Comparison of Post-trained Pruned Models

• LaCo saves post-training computation: 88 GPU*hours versus 216 GPU*hours for LLM-Pruner at 55% sparsity on Llama2-7B.Source: §D.3 · p. 15, Table 21 · p. 15

D.4 Performance Percentage for Main Results

• LaCo-pruned models maintain performance above 70% on most benchmarks and avoid crashes, with no benchmark falling below 30% of Dense performance.Source: Table 22 · p. 16

D.5 Examples of Responses

• On the XSum example, only LaCo generates a smooth, meaningful response according to instructions, while baseline pruners produce repetitive and meaningless text.Source: Table 23 · p. 16

• Detailed scores compare pruned, post-trained, and re-pruned models across all benchmarks.Source: Table 24 · p. 17
• Performance remains stable from 10–25% pruning and is about 70% even at 50% pruning, although it decreases as the pruning ratio increases.Source: Table 25 · p. 17