PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Seungmin Oh, Donggeon Lee, Jongbin Ryu

arXiv:2609.06974v1cs.CLcs.AI

TL;DR

Structured pruning can reduce LLM deployment costs, but compact recovery modules may lack the capacity to reconstruct transformations removed by pruning. OverRep trains an overcomplete recovery module and algebraically merges it into the compact architecture; across multiple backbones, the paper reports improved performance without increased inference-time overhead.

  • Problem

    A compact recovery block may lack the capacity to approximate the complex transformations removed by pruning, limiting recovery as structured pruning reduces deployment costs.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    OverRep temporarily overparameterizes a recovery module during training and algebraically folds it into the compact architecture after training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across multiple backbones, OverRep improves performance without increasing inference-time overhead.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Training-time overcomplete parameterization is presented as a deployment-friendly strategy for structured LLM pruning recovery.

    Source: §5 · p. 9

    The main results focus on layer pruning; attention-head and mixed structured pruning remain unexplored.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.

1 Introduction

Structured pruning reduces LLM deployment costs, but recovery can lack the capacity to reconstruct knowledge removed by pruning. OverRep expands recovery capacity during training, then merges the added components into a compact model; the paper reports improved accuracy and throughput without proportional increases in memory or TFLOPs.

1 Introduction

  • LLMs’ large parameter counts impose substantial memory, latency, and energy costs during deployment.Structured pruning reduces these costs by removing architectural components, but recovery remains comparatively underexplored.Source: §1 · p. 1
  • Capacity-knowledge asymmetry arises when conventional recovery updates only a small fraction of parameters despite pruning removing meaningful knowledge.The mismatch is especially severe when multiple transformer blocks are pruned and a compact module must reproduce their compositional mapping.Source: §1 · pp. 1–2
  • The paper identifies limited recovery capacity as a bottleneck in reconstructing knowledge removed by structured LLM pruning.OverRep’s proposed response is to train with greater recovery capacity and deploy without inference-time overhead.Source: §1 · p. 2
  • OverRep temporarily expands recovery capacity during fine-tuning, then folds the overcomplete module into the compact inference architecture.The method uses an annealed activation for nonlinear training dynamics that converge to a linear regime for exact algebraic merging.Source: §1 · p. 2
  • Larger recovery capacity improves accuracy and throughput without proportionally increasing memory or TFLOPs.The authors identify capacity-knowledge asymmetry as the recovery bottleneck their approach targets.Source: §1 · p. 2

2 Related Work

Prior work studies structured pruning criteria and recovery methods, while re-parameterization expands training-time topology and merges it for inference. OverRep explores greater recovery capacity through a transformer-oriented factorization and annealed activation.

2.1 Structured Pruning

  • Structured-pruning research includes criteria for coupled structures, depth reduction, block removal, sub-layer pruning, and embedding-dimension reduction.The paper’s empirical evaluation primarily considers block- or layer-level pruning.Source: §2.1 · p. 2
  • Most recovery methods use LoRA-based fine-tuning, while others use layer-wise distillation; LoRA’s low-rank parameterization remains a capacity constraint.RankAdaptor and RestoreLCC adapt LoRA to reconstruction, and Streamline and CoMe use layer-wise distillation.Source: §2.1 · pp. 2–3
  • OverRep complements pruning criteria and recovery objectives by increasing expressive capacity during recovery, then folding it into a compact module.The comparison holds the pruning mask and distillation target fixed while emphasizing the recovery module’s ability to approximate removed transformations.Source: §2.1 · p. 3

2.2 Re-parameterization

  • Prior re-parameterization work uses richer training-time topologies that are algebraically merged into a single inference structure.The passage describes developments from convolutional networks to vision transformers and block-level skip connections.Source: §2.2 · p. 3
  • OverRep uses a cascaded multiplicative-additive factorization for projection-dominated transformer blocks and anneals its activation back to a linear regime for exact merging.Its recovery-specific choices include warm-up to preserve the pretrained operating point and selective placement within frozen recovery blocks.Source: §2.2 · p. 3

3 Method

OverRep addresses the mismatch between the knowledge removed by pruning and the limited capacity of a compact recovery block. It expands recovery capacity during training, then re-parameterizes the module for compact deployment while using an annealed activation to preserve exact merging.

3.1 Preliminaries

  • Standard layer-wise distillation trains one retained block to map the hidden state before a pruned segment to the state after it, which can exceed that block’s capacity.The mismatch is especially relevant when one block must approximate the complex nonlinear relationship across a segment of original blocks.Source: §3.1 · pp. 3–4

3.2 Overcomplete Re-parameterization for Scaling Recovery Capacity

  • OverRep trains an overcomplete recovery module and algebraically folds it into the original compact architecture without additional inference-time parameters, memory, or latency.The training objective may freeze θ or jointly fine-tune θ and ϕ; experiments use a two-block instantiation.Source: §3.2 · p. 4

3.3 Overcomplete Recovery Module

  • The ORM expands each projection with auxiliary parameters W_i and D_i, initialized so recovery begins at the pretrained operating point.W_i adds a parallel branch, while D_i applies a cascaded transformation in the output space.Source: §3.3 · p. 4
  • An annealed activation supports nonlinear recovery, then returns to the identity map during final linear stabilization before re-parameterization.The schedule starts with α_0 = 0, introduces nonlinear dynamics during warm-up, and keeps α_s = 0 for the final stabilization phase.Source: §3.3 · pp. 4–5
  • OverRep changes the optimization geometry of the compact deployed weight through a parameter-dependent, positive-semidefinite preconditioning of the gradient.The analysis is limited to a frozen-P_i, continuous-gradient-flow model in the final linear phase and does not model AdamW or nonlinear activation annealing.Source: §3.3 · p. 5

4 Experiments

The experiments compare OverRep with recovery methods and complete pruning pipelines across reasoning and generation benchmarks, then examine its efficiency, components, and capacity-related behavior. Results favor OverRep across these settings, while its training-time capacity increase does not translate proportionally into higher training cost.

4 Experiments

  • The study evaluates LLaMA2, LLaMA3, and Qwen3 against pruning and recovery baselines, using standardized FineWeb-Edu recovery data and lm-eval-harness evaluation.Recovery uses 120,000 training samples and 4,000 test samples.Source: §4 · p. 5

4.1 Training Efficiency and Resource Analysis

  • OverRep keeps memory and TFLOPs comparable to recovery baselines; OverRep* cuts peak GPU memory by up to 1.6× and TFLOPs by 2.4×, while raising throughput by up to 2.8× over vanilla OverRep.OverRep also achieves higher throughput and stronger reasoning accuracy than existing recovery baselines on both backbones.Source: Figure 3 · p. 6, §4.1 · p. 6

4.2 Reasoning Performance

  • OverRep beats the strongest controlled recovery baseline by up to 5.5 points at 25% pruning and 8.4 points at 50%, and leads in every controlled setting.The comparison uses average retained performance (RP) on reasoning tasks.Source: §4.2 · p. 6, Table 1 · p. 7
  • OverRep achieves the strongest average reasoning RP across complete pruning pipelines, backbones, and pruning ratios.The comparison includes methods using their own pruning and recovery procedures.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Generation Performance

  • OverRep outperforms controlled recovery baselines in generation across all backbones and pruning ratios, and has the best average RP against complete pipelines on all backbones at 25% pruning and five of six at 50%.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Ablation Study

  • The hybrid ablation with frozen-block partial application and annealed activation performs best; MLP-side components outperform attention-only components, and Wi and Di work complementarily.The ablation uses LLaMA3-3B at 25% pruning.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 Analysis

  • Under an iso-parameter budget, high-rank LoRA does not close OverRep’s performance gap and requires up to 2.9× more GPU-hours.The comparison matches LoRA to approximately 650M trainable parameters under the same protocol.Source: Table 5 · p. 8, §4.5 · p. 9
  • Full fine-tuning trails OverRep in reasoning at both pruning ratios despite higher cost, while OverRep-KD outperforms the same logit-KD objective with less than one-quarter of its GPU-hours.The results indicate benefits beyond exposing more trainable parameters and complementarity between OverRep and logit-level distillation.Source: §4.5 · p. 9, Table 6 · p. 9

5 Conclusion

OverRep addresses capacity-knowledge asymmetry in structured LLM pruning by training with an overcomplete recovery module and folding it back into the compact architecture. Across multiple backbones, it improves performance without increasing inference-time overhead.

5 Conclusion

  • OverRep temporarily expands recovery capacity during training, then algebraically re-parameterizes the module into the compact inference architecture.Its annealed activation supports nonlinear training dynamics before reaching a linear regime that permits merging.Source: §5 · p. 9
  • Across multiple backbones, OverRep improves performance without increasing inference-time overhead.The analysis also reports that greater recovery capacity does not necessarily require a proportional increase in training cost.Source: §5 · p. 9
  • The results support training-time overcomplete parameterization as a deployment-friendly strategy for structured LLM pruning.The strategy preserves the compact inference-time architecture through algebraic re-parameterization.Source: §5 · p. 9

Limitations

OverRep’s evaluation is centered on layer pruning, while several pruning granularities and settings remain preliminary or untested. Its cached training variant also depends on a reusable frozen prefix, and its benefits may vary across settings.

Limitations

  • Main results use layer pruning; channel-wise and hybrid results remain preliminary, while attention-head and mixed structured pruning are unexplored.The preliminary results use smaller recovery budgets and cover only one or two backbones; broader evaluation across pruning granularity is future work.Source: Limitations · p. 10
  • OverRep increases trainable parameters, and cached recovery assumes a frozen, reusable prefix, so its benefits may vary with pruning patterns, hardware, and training implementations.The paper notes that feature caching can further accelerate recovery.Source: Limitations · p. 10

Appendix

  • In the appendix, ‘Rea.’ and ‘Gen.’ denote average raw scores over 10 reasoning and 3 generation tasks, not retained performance (RP).Source: Appendix · p. 12

A Practical Implementation

The implementation uses two recovery blocks after the pruned segment, then merges their overcomplete projections into compact transformer blocks. Experiments standardize data and evaluation where possible and report reproduction checks and confidence intervals.

A Practical Implementation

  • The practical implementation uses two consecutive recovery blocks after the pruned segment, keeping the active training path short.It jointly optimizes the first block’s pretrained and auxiliary parameters, freezes the second block’s pretrained parameters, and optimizes its auxiliary parameters; both blocks are merged after training.Source: A Practical Implementation · p. 12

B Implementation Details

B.1 Recovery Dataset

  • Recovery uses 120,000 FineWeb-Edu training samples and 4,000 test samples, tokenized to sequence length 1024 with a fixed random seed of 42.The same dataset configuration is used across backbones and methods.Source: §B.1 · p. 13

B.2 Training Configurations

  • Training starts from each baseline’s official configuration, with limited hyperparameter searches; LaCo uses LoRA, and Streamline’s epochs are increased for stable recovery.The study also reports shared optimizer settings and adjustments to make training comparisons feasible in its hardware setting.Source: §B.2 · p. 13

B.4 Confidence Intervals

  • Confidence-interval halfwidths are reported for pruned LLaMA3-3B on reasoning and generation benchmarks, computed over three random seeds.Each table entry c corresponds to a confidence interval of ±c.Source: Table 10 · p. 14, §B.4 · p. 14

B.3 Reproduction Verification

  • Reproduced Streamline results closely match the original paper’s values under its evaluation framework, supporting baseline-reproduction fidelity.Source: §B.3 · p. 14, Table 9 · p. 14

B.5 Recovery Training Fairness

The fairness analyses compare OverRep under both matched wall-clock budgets and a shared 10-epoch schedule. The authors report that its gains persist under both comparisons.

B.5 Recovery Training Fairness

  • The comparison uses method-specific recovery epochs to make wall-clock training time comparable rather than equalizing training tokens.Source: §B.5 · p. 14
  • Despite having the most trainable parameters, OverRep’s wall-clock recovery time is comparable to or lower than existing baselines for LLaMA3-3B after 25% pruning.Including cached-feature precomputation, which takes less than one hour, leaves Streamline* and OverRep* total recovery time comparable.Source: Table 11 · p. 14, §B.5 · p. 14
  • Under a shared 10-epoch recovery schedule, OverRep still outperforms the baselines on LLaMA3-3B reasoning and generation benchmarks.This comparison controls recovery epochs and samples seen during training.Source: §B.5 · pp. 14–15, Table 12 · p. 15

B.6 Training-cost Quantification

Training-cost analyses compare OverRep with LoRA and with otherwise identical Plain recovery. The appendix also adapts OverRep to channel-wise pruning and reports results under a smaller-budget LLM-Pruner comparison.

B.6 Training-cost Quantification

  • Including cache precomputation, OverRep is 6.1× cheaper on Qwen3-14B and 34× cheaper on Qwen3-30B-A3B than LoRA.LoRA requires eight RTX 3090 GPUs on the two larger backbones, while OverRep recovery runs on one GPU.Source: §B.6 · p. 15

C.1 OverRep in Channel-wise Pruning

  • For channel-wise pruning, OverRep adapts the shapes of Wi and Di to the reduced intermediate width and merges projections into standard reduced-width projections.The block input and output dimensions remain unchanged, and deployment adds no parameters or computation.Source: §C.1 · pp. 15–16

B.6 Training-cost Quantification

  • Overcomplete parameterization increases recovery time by 2.4–6.6× over Plain recovery; the largest run requires 12.6 single-GPU hours.Plain recovery uses the same localized blocks and 20 epochs without auxiliary Wi and Di components or annealed activation.Source: §B.6 · p. 15, Table 14 · p. 16

C.1 OverRep in Channel-wise Pruning

  • On Q3-30B, OverRep reports 53.1 Rea., 30.1 Gen., and 8.8h, compared with Streamline’s 49.5 Rea., 22.2 Gen., and 7.3h.LoRA reports 49.8 Rea., 23.4 Gen., and 53.3h × 8 GPUs.Source: §C.1 · p. 16, Table 16 · p. 16
  • With LLM-Pruner’s protocol and a 24K-sample, 5-epoch recovery budget, OverRep improves the average by 5.6 points on LLaMA3-3B and 4.2 points on LLaMA3-8B.These results use nine reasoning tasks without the generation suite and are not directly comparable to the main protocol’s 120K samples for 20 epochs.Source: §C.1 · p. 16, Table 15 · p. 16

C.2 OverRep in Various Architectures

OverRep is evaluated on larger, MoE, and hybrid architectures, and across existing pruning criteria, with reported benefits in recovery efficiency and performance.

C.2 OverRep in Various Architectures

  • On Qwen3-30B-A3B and Qwen3-14B, OverRep outperforms LoRA with 48× and 7.2× fewer aggregate GPU-hours, respectively.Recovery is layer-local, with training memory bounded by two recovery blocks plus cached features; the router remains untouched, preserving expert routing and sparsity at deployment.Source: §C.2 · p. 16
  • At 50% pruning on Nemotron-H-4B, OverRep improves reasoning by 3.2 points, while OverRep-KD improves generation by 4.6 points.At 25% pruning, OverRep modestly improves reasoning at less than half the cost, LoRA remains stronger on generation, and OverRep-KD improves both metrics; OverRep is applied to attention and MLP sublayers while the Mamba-2 mixer stays frozen.Source: §C.2 · pp. 16–17, Table 17 · p. 17

C.3 Compatibility with Existing Pruning Criteria

  • Across eight settings using contiguous Streamline and non-contiguous ShortGPT masks, OverRep improves reasoning in six and generation in seven, with remaining differences below one point.The reported gains require no criterion-specific tuning.Source: §C.3 · p. 17, Table 18 · p. 17

D Detailed Experimental Results

The detailed-results section organizes reasoning and generation evaluations across LLaMA2, LLaMA3, and Qwen3, reporting task-specific metrics and relative performance.

D.1 Reasoning Performance

D.2 Generation Performance

Loading 2609.06974v1…