Source-linked AI summary
On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim, Keith Ross
TL;DR
Layer pruning can preserve classification performance, but the paper investigates why generative reasoning remains harder to recover under constrained post-training. It evaluates self-generated-response recovery and complementary stress tests, finding persistent reasoning deficits linked to arithmetic and syntactic capabilities. The results characterize practical limits and task dependence while keeping the scope bounded by the tested recovery regimes.
Problem
Prior work provides limited characterization of pruning-induced generative reasoning failures, while reported recovery methods often require large-scale data and compute.
Method
The paper evaluates self-generated-response finetuning under a single-80GB-GPU budget, supplemented by task-aligned full finetuning, RLVR, and a Minitron case study.
Results
Classification recovers more readily than generative reasoning, with the gap persisting across the evaluated recovery settings and linked to arithmetic and syntactic deficits.
Takeaways & Limitations
Layer pruning is strongly task-dependent: it may remain useful for classification and less reasoning-intensive tasks, while aggressive depth reduction threatens generative reasoning.
Takeaways & Limitations
The study does not exhaustively scale all recovery methods, and its Minitron case study differs from the controlled experiments in pruning and training conditions.
Abstract
from arXiv · showhide
Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative reasoning tasks, such as GSM8K and HumanEval\textsuperscript{+}, exhibit substantially weaker recovery. We show that beyond surface-level text degradation, pruning leads to a loss of key algorithmic capabilities, including arithmetic computation and balanced parenthesis generation. Under realistic post-training constraints, using a single 80GB GPU and without access to pretraining-scale data or compute, we evaluate a simple recovery strategy based on supervised finetuning with self-generated responses. This approach recovers up to 90\% of baseline performance on classification tasks, but recovery for generative reasoning remains limited. We further find that this gap persists even under a favorable task-aligned recovery setting, where pruned models are fully finetuned on self-generated GSM8K responses, suggesting that the degradation is not merely due to generic instruction data or parameter-efficient tuning. As complementary evidence, we analyze a depth-pruned model trained with nearly 100B post-pruning tokens and find that deficits persist even on simple arithmetic tasks that do not require multi-step generation. Overall, we characterize practical recovery limits of layer pruning for generative reasoning and provide guidance on when depth reduction is effective under constrained post-training regimes.
1 Introduction
Layer pruning is attractive for efficient LLM compression, but this paper examines why generative reasoning recovers more poorly than classification under realistic post-training constraints. It evaluates self-generated-response recovery and identifies persistent algorithmic deficits beyond surface-level text degradation.
- 1 Introduction: Layer pruning reduces model depth with minimal or no finetuning, motivating its use for efficient LLM compression.The paper frames layer pruning as a simple alternative to finer-grained compression methods.
- 1 Introduction: Existing recovery methods for generative tasks often require billions of tokens and substantial compute, motivating evaluation under a single ∼80GB GPU.The paper targets moderate-scale post-training data without pretraining-scale resources.
- 1 Introduction: Self-generated responses provide a practical baseline for measuring how much capability pruned models recover without pretraining-scale data.The approach uses self-generated supervision to isolate post-pruning recovery under constrained resources.
- 1 Introduction: The generative reasoning recovery gap persists with self-generated responses across model families despite strong improvements on classification benchmarks.This comparison is the paper’s central evaluation result under realistic post-training constraints.
- 1 Introduction: Pruning disrupts arithmetic computation and syntactic structure, including parenthesis tracking, beyond surface-level text degeneration.These capabilities are identified as failure modes relevant to multi-step reasoning.
- 1 Introduction: A depth-pruned model trained with nearly 100B post-pruning tokens still exhibits reasoning and arithmetic deficits.The Minitron analysis supplies complementary evidence outside the controlled recovery setup.
- 1 Introduction: The study aims to clarify when layer pruning succeeds or fails and guide its use when preserving generative reasoning is a priority.Its stated goal is characterization and practical guidance rather than a new pruning algorithm.
2 Layer Pruning and Recovery
The paper evaluates layer-pruning recovery under constrained post-training and finds that self-generated supervision improves retention, especially for classification, while generative reasoning remains substantially harder to recover. Task-aligned full finetuning and complementary evidence show that this gap persists beyond generic instruction data and QLoRA.
- Recovery setup: The pipeline removes layers using Block Influence or Reverse Order and then recovers performance through post-training.Experiments use QLoRA on a single 80GB GPU, with comparisons to LoRA and full finetuning.
- Recovery setup: Self-generated responses consistently outperform standard finetuning on fixed open-source datasets after pruning.SGR discards reference responses, uses outputs from the unpruned model, and finetunes the pruned model on the resulting pairs.
- 2.1 Results: Classification typically retains close to or above 80%, whereas generative performance remains below 65% in most cases.For the same LLaMA model, classification retention is nearly 90% while generative retention is 63.4%.
- 2.2 Task-Aligned Recovery on GSM8K: Full finetuning on self-generated GSM8K responses improves pruned models but leaves them below unpruned baselines across LLaMA, Mistral, and Qwen.This task-aligned setting uses the GSM8K training split and full-parameter finetuning, making it more favorable than the main QLoRA experiments.
- 2.3 Case Study on Minitron: The Minitron case study provides suggestive external evidence that deficits persist despite approximately 94B post-pruning distillation tokens.Because its pruning and training setup differs from the controlled experiments, the authors treat this evidence cautiously.
- Scope and limitations: The study is scoped to resource-constrained recovery and does not establish that layer-pruned models are impossible to recover.The authors state that the observed gap cannot be attributed solely to QLoRA, generic instruction data, or the single-GPU constraint.
3 Layer-by-Layer Pruning for Generative Tasks
Single-layer pruning reveals task-specific failures: reasoning and coding benchmarks can collapse at particular layers even when summarization remains comparatively robust, and the losses extend beyond surface text degeneration to arithmetic and syntax.
- Layer-by-Layer Setup: Single-layer pruning was evaluated across GSM8K, HumanEval+, and XSUM on three instruction-tuned model families.The analysis removes one transformer layer at a time to diagnose generative-task failures.
- Text Degeneration: Text degeneration is measured with 4-gram repetition and Self-BLEU4, with higher values indicating more repetition and less diversity.These metrics quantify repetitive, less diverse generation after pruning.
- Text Degeneration: Text degeneration does not fully explain reasoning failures: sharp math and coding drops can occur with intact generation quality, while Mistral shows degeneration without a corresponding task-performance drop.Removing Mistral layer 24 causes rambling after a valid response.
- Cross-Benchmark Effects: Reasoning-intensive GSM8K and HumanEval+ show severe degradation at specific layers, whereas XSUM is comparatively robust except where text degeneration occurs.Layer 0 is generally skipped because of poor results.
- Degradation of Arithmetic: Pruning impairs arithmetic itself, including simple calculations and first-token arithmetic accuracy without any generation requirement.Middle-region layer removals produce substantial log-probability and accuracy drops across 200 arithmetic problems.
- Degradation of Parenthesis Tracking: Pruning also disrupts syntactic consistency, especially parenthesis tracking, with certain layers sharply increasing unbalanced-parenthesis and other invalid-syntax errors.Qwen layer 23 produces a pronounced increase in parenthesis-matching errors; Mistral also shows malformed code-block failures.
- Interpretation: Single-layer results are diagnostic rather than predictive and identify candidate failure modes for later multi-layer pruning and recovery analysis.The study does not assume that multiple-layer effects compose linearly.
4 Post-Recovery Analysis
Post-pruning recovery improves arithmetic and coding behavior but leaves substantial gaps relative to base models, including on simple arithmetic and syntactic validity.
- Arithmetic Ability: At a fixed 25% pruning ratio, pruning reduces average arithmetic accuracy to 29%, while SGR finetuning only partially recovers the loss.Finetuned pruned models remain well below their base models.
- Arithmetic Ability: Even simple arithmetic without multi-step reasoning or long-form generation retains a gap between base and finetuned pruned models.This indicates that recovery difficulty is not limited to complex mathematical generation.
- Arithmetic Ability: Minitron arithmetic accuracy falls from 58% for the base LLaMA-3.1-8B model to 13% after pruning and remains unrecovered after nearly 100B post-pruning tokens plus SGR finetuning.The result supports persistent arithmetic deficits after extensive post-pruning training.
- Code Generation: After finetuning, pruned models continue to struggle with syntactically valid code, especially on MBPP+ compared with HumanEval+.MBPP+ requires code generation from natural-language descriptions, whereas HumanEval+ supplies a function signature prefix.
- Code Generation: Persistent undefined-variable and unbalanced-parenthesis errors show that finetuning only partially restores syntactic and state consistency during generation.Invalid and logically incorrect code remains prevalent across models.
- Code Generation: Higher syntactic validity does not guarantee functional correctness, as illustrated by Qwen on HumanEval+.The fraction of functionally correct solutions remains substantially lower than the syntactically valid fraction.
5 Background & Related Work
Prior work motivates layer pruning through layer redundancy and strong classification retention, but generative reasoning remains less consistently preserved and its failure modes are insufficiently characterized.
- Layer Pruning in LLMs: Layer pruning removes entire transformer blocks to reduce model depth, often with minimal or no additional finetuning.Its appeal is supported by theoretical and empirical claims of layer redundancy.
- Layer Pruning in LLMs: Classification benchmarks retain over 80% of baseline accuracy after removing 20–25% of layers in prior studies.These results helped establish layer pruning as an effective compression strategy for classification.
- Layer Pruning in LLMs: Those gains do not consistently extend to generative reasoning tasks such as GSM8K, which require multi-step generation.The discrepancy suggests that layer redundancy depends on the task.
- Limitations for Generative Reasoning: Existing accounts emphasize deeper layers for reasoning, while systematic analysis of pruning-induced failure modes and their persistence after recovery remains limited.The paper addresses this gap under practical post-pruning constraints and diagnoses arithmetic and structured-generation failures.
6 Conclusion
Across evaluated recovery settings, layer pruning is more compatible with classification than generative reasoning because arithmetic and syntactic losses persist beyond surface generation quality.
- Conclusion: Across SGR, task-aligned full-parameter finetuning, and RLVR, pruned models recover classification more readily than generative reasoning.Minitron provides complementary evidence for this recovery gap.
- Conclusion: Persistent arithmetic and syntactic losses, rather than only surface-level generation quality, are linked to the generative-reasoning recovery gap.The diagnostic findings identify arithmetic computation and structured generation as enduring deficits.
- Conclusion: Moderate pruning may remain useful for classification and less reasoning-intensive tasks, whereas aggressive depth reduction can severely impair reasoning.The conclusion frames pruning as strongly task-dependent.
- Conclusion: Preserving reasoning capabilities may require multistage training or large-scale compute beyond resource-constrained setups.This boundary is stated for the settings considered by the paper.
Limitations
The study is scoped to practical post-pruning recovery and does not exhaustively evaluate all recovery methods or forms of generative reasoning.
- The primary experiments use moderate-scale instruction data, limited compute, and QLoRA-based finetuning.Additional checks include task-aligned full finetuning, RLVR, and a Minitron case study.
- The Minitron case study is complementary rather than a clean causal comparison because its pruning and post-training settings differ.Differences include pruning ratio, layer selection, training data, and post-training pipeline.
- The failure-mode analysis focuses on arithmetic and parenthesis or syntax tracking rather than exhaustive generative reasoning.The paper identifies broader structured generation and targeted recovery as directions for future work.
A.1 Minitron
The paper analyzes Llama-3.1-Minitron-4B-Depth as complementary evidence, finding persistent degradation after substantial post-pruning distillation and additional self-generated-response finetuning.
- A.1 Minitron: Llama-3.1-Minitron-4B-Depth is a 50%-depth-pruned model trained by knowledge distillation on approximately 94B tokens.Contiguous layers were selected for removal based on minimal accuracy loss, using the unpruned model as teacher.
- A.1 Minitron: 23.9% accuracy was achieved after further SGR finetuning, while performance remained substantially degraded despite the much larger post-pruning training budget.The result is presented as suggestive external evidence rather than a clean causal comparison.
- A.1 Minitron: The released Minitron checkpoint was evaluated before the additional math, code, instruction, and preference-alignment stages reported by prior work.The authors therefore finetune it with self-generated responses before evaluation.
- A.1 Minitron: The findings suggest that strong downstream performance may require large-scale distillation together with curated data and multi-stage alignment.This supports examining recovery under settings where those resources may be unavailable.
A.2 Tokens with Layer Pruning
Figure 7 measures how single-layer pruning changes average generated response length relative to the unpruned baseline.
- A.2 Tokens with Layer Pruning: The figure’s quantity is average generated response length.
- A.2 Tokens with Layer Pruning: The intervention is single-layer pruning.
- A.2 Tokens with Layer Pruning: The comparison reference is the unpruned baseline.
A.3 Arithmetic Mistake
The weekly egg-production example presents a pruned-model solution that progresses from the chicken count to daily eggs and then to a weekly total of 2448 eggs.
- A.3 Arithmetic Mistake: The example asks for Gary’s current weekly egg collection after his chicken count increases eightfold.Gary initially buys 4 chickens, and each chicken lays 6 eggs per day.
- A.3 Arithmetic Mistake: The pruned-model response identifies the initial chicken count and the eightfold increase before continuing the calculation.
- A.3 Arithmetic Mistake: The response states that the daily egg total is computed from the number of chickens and the per-chicken laying rate.The stated rate is 6 eggs per chicken per day.
- A.3 Arithmetic Mistake: The weekly calculation multiplies the daily egg number by 7.
- A.3 Arithmetic Mistake: 2448 eggs per week is the response’s stated conclusion.
A.4 Arithmetic Ablation Experiment Details
The arithmetic ablation restricts outputs to single digits and examines how pruning affects arithmetic ability and generative behavior. The broader experiments compare recovery methods, pruning strategies, efficiency gains, and reasoning degradation under post-training constraints.
- Arithmetic Ablation: The arithmetic experiment uses the single-digit, three-operation subset of EleutherAI/arithmetic and restricts outputs to digits 0–9.
- Arithmetic Ablation: Single-layer pruning is evaluated for its effect on the arithmetic ability of various models.
- Generative Failure Modes: A code-generation example illustrates pruning-related syntactic failure through a parenthesis mismatch in the generated program.
- Recovery Experiments: QLoRA recovery is comparable to full-parameter finetuning despite substantially lower memory and compute requirements.The comparison uses the same iteratively pruned Qwen model with seven layers removed and evaluates GSM8K as a proxy for generative reasoning quality.
- Recovery Experiments: RLVR raises Qwen’s GSM8K retention from 0.329 after SGR to 0.418, but gains do not close the gap with unpruned models and may become unstable with longer training.For Mistral, the best RLVR checkpoint reaches 28.51% GSM8K accuracy before performance declines sharply with continued optimization.
- Efficiency–Reasoning Trade-off: Removing two layers retains approximately 85–90% of original performance while increasing throughput by 1.1–1.15×, whereas more aggressive pruning disproportionately harms generative reasoning.At approximately 25% layer removal, throughput rises around 1.3× and GPU memory falls roughly 20–25%, but reasoning degradation is larger.