Source-linked AI summary
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Pietro Moriello, Pietro Buzzega, Angelo Porrello, Simone Calderara
TL;DR
Resource budgets vary, but structured pruning usually targets a predefined size and slimmable capability is typically learned from scratch, leaving a gap for pretrained LLMs. IREKOGPT extends SliceGPT’s ordered PCA structure to nested widths and calibrates and corrects across widths; across models and sparsity levels, it improves over PCA-only slimming, with further gains at higher compression.
Problem
Resource availability can vary, while structured pruning targets a predefined size and slimmable models are typically trained from scratch, creating a gap for pretrained LLMs.
Source: §1 · p. 1Method
IREKOGPT retains SliceGPT’s full ordered PCA projections for nested widths, calibrates across multiple compression ratios, and corrects weights using full-width outputs as targets.
Source: §3.1 · p. 2, §3.2 · pp. 2–3, §3.3 · p. 3Results
Across models and sparsity levels, ECD improves over PCA-only slimming, and at higher compression IREKOGPT improves further, sometimes matching or outperforming SliceGPT.
Source: §4 · p. 4Takeaways & Limitations
IREKOGPT improves robustness across compression levels while retaining a single dynamically configurable model.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We introduce IrekoGPT, a post-hoc method for converting pretrained LLMs into slimmable models whose width can be adjusted at inference time. Building on SliceGPT, we retain its projection matrices without pruning them, allowing a single model to expose nested subnetworks at different widths. We improve robustness by calibrating each layer across multiple compression ratios, and correct downstream linear layers through gradient-free ridge regression. Across Llama and Qwen models, preliminary results show improvements over naive PCA-based slimming, with the largest gains at high compression. Code is available at https://github.com/aimagelab/IrekoGPT
1 Introduction
Structured pruning can produce compact models, but fixed target sizes and the cost of maintaining separate compressed networks do not fit changing resource budgets; slimmable models address width choice but are typically trained from scratch. IREKOGPT applies slimmable inference to pretrained LLMs by extending SliceGPT and reports particularly pronounced gains at higher compression.
1 Introduction
- Structured pruning can save resources without dedicated hardware support by slicing weight matrices, while compressed networks retain most original functionality and are lighter to store and faster.Other approaches merge similar neurons or use eigenvector decompositions to select components for removal.Source: §1 · p. 1
- Changing resource availability makes fixed-size pruning awkward: separate compressed networks for different budgets add storage costs and loading and unloading overhead, while slimmable models are typically learned from scratch.The paper frames this as a mismatch with growing reliance on pretrained foundation models.Source: §1 · p. 1
- IREKOGPT addresses converting pretrained LLMs into slimmable variants by extending SliceGPT, with particularly pronounced gains reported at higher compression.The approach modifies alignment between smaller-subnetwork and full-width activations.Source: §1 · p. 1
2 Background
SliceGPT uses layer activations from calibration text to compute an ordered PCA basis, then retains leading components to reduce width. Its transformations are applied sequentially, but only to selected dimensions of linear layers adjacent to LayerNorm.
2 Background
- At each layer, SliceGPT extracts activations from calibration text and computes a PCA eigenvector matrix ordered by decreasing eigenvalue, retaining its first d columns for a reduced projection.The activations have dimensions for examples, sequence positions, and token dimension.Source: §2 · p. 2
- The projections are absorbed into model weights before slicing; in this paper, slicing means retaining the first d columns of each projection matrix.The passage refers readers to Ashkboos et al. [2024] for the detailed weight operations.Source: §2 · p. 2
- SliceGPT processes layers sequentially, so pruning at earlier layers shifts downstream activations and affects the principal components computed later.Each step’s PCA reflects distributional shifts induced by preceding pruning.Source: §2 · p. 2
- The procedure changes only input or output dimensions of linear layers adjacent to LayerNorm; other dimensions remain at full width.Source: §2 · p. 2
3 Method
IREKOGPT turns SliceGPT’s ordered PCA projections into nested widths, then calibrates across width configurations and corrects weights using full-width outputs as targets. The correction jointly handles the considered compression ratios and has a closed-form least-squares solution.
3.1 Revisiting SliceGPT: from slicing to slimming
- Absorbing the full ordered eigenvector matrix without slicing lets each projection be truncated to different widths, producing nested subnetworks that retain the highest-variance components.This repurposes SliceGPT’s ordered PCA basis for dynamic width selection at inference time.Source: §3.1 · p. 2
3.2 Improving slimmability by expanding the calibration dataset
- ECD addresses width-dependent activation shifts by propagating parallel streams at the considered compression ratios and jointly using their concatenated activations to compute each PCA basis.The basis reflects all considered widths rather than only full-width activations; cached activations reduce computation during iterative compression.Source: §3.2 · pp. 2–3
3.3 Leveraging full-width signals for slim subnetworks
- A residual correction to the original weight matrix fits compressed-stream inputs to full-width target outputs, with regularization and a joint objective across compression ratios.The least-squares objective admits an efficient closed-form solution.Source: §3.3 · p. 3
4 Experiments
Across the evaluated models and compression levels, expanded calibration improves on PCA-only slimming, while IREKOGPT often improves further at higher compression. Its value-projection outputs also more closely match the full-width model, especially as sparsity increases.
4 Experiments
- The setup evaluates four Llama and Qwen models using 128 WikiText-2 calibration sequences and expanded-calibration compression ratios of 0.1, 0.2, and 0.3.The ridge correction targets the value projection and uses 65,536 sampled tokens per layer with λ = 1.Source: §4 · p. 3
- Evaluation measures WikiText-2 perplexity, with lower values indicating better language-model performance.Sequences are truncated or concatenated to a maximum length of 2048.Source: §4 · p. 3
- Expanded calibration consistently improves performance over PCA-only, and IREKOGPT further improves performance at higher compression, sometimes matching or outperforming SliceGPT; Qwen3-1.7B is the main exception.SliceGPT is not slimmable and requires a separate compression procedure for each target sparsity.Source: §4 · p. 4, Table 1 · p. 4
- The representation-level comparison shows IREKOGPT consistently reduces value-projection MSE versus PCA-only and ECD, with the gap especially evident as sparsity increases.The metric compares compressed and full-width value-projection outputs; lower MSE is better.Source: §4 · p. 4, Figure 1 · p. 4
5 Conclusion
IREKOGPT reinterprets SliceGPT’s PCA structure to create post-hoc slimmable pretrained LLMs. Calibrating across widths and using full-width signals improves robustness across compression levels while retaining one dynamically configurable model.
5 Conclusion
- IREKOGPT turns pretrained LLMs into slimmable models by reinterpreting SliceGPT’s PCA structure.The approach is post-hoc and retains a single model that can be configured dynamically.Source: §5 · p. 4
- The method calibrates across multiple widths and leverages full-width signals to improve robustness across compression levels.Source: §5 · p. 4
- The resulting model retains dynamic width configuration in a single checkpoint.Source: §5 · p. 4
A Additional metrics plots.
The additional plots compare representation-level metrics between compressed and full-width value-projection outputs across models and layers. They show layer-wise CKA and MSE, including differences between ECD and IREKOGPT relative to PCA-only.
A Additional metrics plots
- The appendix presents additional plots to assess the method across a wider range of models and layers.Source: A Additional metrics plots · p. 6
- Figure 2 compares layer-wise CKA between compressed and full-width value-projection outputs across increasing sparsity; higher is better.It shows all models and layers and compares PCA-only, ECD, and IREKOGPT.Source: Figure 2 · p. 6
- Figure 3 compares layer-wise MSE between compressed and full-width value-projection outputs across increasing sparsity; lower is better.It shows all models and layers and compares PCA-only, ECD, and IREKOGPT.Source: Figure 3 · p. 7
- Figure 4 plots layer-wise CKA differences for ECD and IREKOGPT relative to PCA-only.CKA is computed between compressed and full-width value-projection outputs.Source: Figure 4 · p. 7
- Figure 5 plots layer-wise MSE differences for ECD and IREKOGPT relative to PCA-only.MSE is computed between compressed and full-width value-projection outputs.Source: Figure 5 · p. 8