PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration-Preserving Pruning: Compression as a Reliability Contract

Ibne Farabi Shihab, Adria Binte Habib, Anuj Sharma

arXiv:2608.23744v1cs.LGcs.CL

TL;DR

The paper asks whether pruning can preserve score geometry well enough to produce smaller valid prediction sets without treating pruning as the source of conformal coverage. It introduces Calibration-Preserving Pruning with disjoint data roles and evaluates efficiency, finding the clearest gains on large-label classification while generic gradients explain much of the improvement.

  • Problem

    Compressed models may retain top-label accuracy yet become less useful when downstream systems consume calibrated probabilities or conformal prediction sets.

  • Method

    Calibration-Preserving Pruning augments base pruning scores with nonconformity-gradient saliency and separates pruning, validation-selection, conformal-calibration, and test data roles.

  • Results

    Across 15 dataset–sparsity cells, CPP-SparseGPT produces smaller sets in 13 and higher accuracy in 11, with the clearest gains on large-label tasks; threshold-aware CPP reaches 7.8 mean set size at explicit accuracy and offline-compute costs.

  • Takeaways & Limitations

    CPP can improve conformal efficiency, especially with threshold-aware saliency, while coverage comes from split independence and accuracy remains a separate utility guardrail.

  • Takeaways & Limitations

    The claims are limited to reliability-sensitive classification, and the coverage guarantee requires exchangeability and strict independence of the final calibration split from pruning and selection decisions.

Abstract

from arXiv · show

Split conformal prediction, not the pruning rule, supplies finite-sample marginal coverage once a pruned model is fixed independently of the conformal calibration split. We study the separate efficiency problem: can pruning preserve score geometry well enough to obtain smaller valid prediction sets? Calibration-Preserving Pruning (CPP) augments a base pruning score with nonconformity-gradient saliency and uses disjoint pruning, validation-selection, conformal-calibration, and test splits. Bounded score perturbations imply bounded conformal-quantile shifts and controlled set inflation, but do not make the generic coverage theorem CPP-specific. Final five-seed Qwen2.5-1.5B results at 50\% sparsity show the largest gains on large-label tasks. On DBpedia-14, CPP-SparseGPT reduces mean set size from \(10.1\) to \(8.6\) while changing accuracy from \(0.347\) to \(0.366\); CPP-Wanda reduces \(11.2\) to \(9.0\) with an accuracy trade-off from \(0.310\) to \(0.295\). Across 15 dataset--sparsity cells, CPP-SparseGPT produces smaller sets in 13 and higher accuracy in 11. Matched controls show that generic supervised gradients explain much of the gain: true-label CPP is not statistically resolved from matched Wanda+SNIP, whereas threshold-aware candidate-label CPP reaches \(7.8\) mean set size at explicit accuracy and offline-compute costs. RoBERTa-base and Llama-3-8B diagnostics support transfer, but our claims remain limited to reliability-sensitive classification.

1 Introduction

The paper distinguishes generic conformal validity from the separate goal of preserving informative prediction sets after pruning. CPP targets score geometry with gradient-based saliency, and controlled experiments narrow its gains to objective-specific efficiency effects.

  • Independent split-conformal recalibration provides finite-sample marginal coverage for any fixed pruned classifier, but CPP does not create that validity result.
  • Pruning can preserve coverage while reducing informativeness when compression collapses separation among plausible labels and forces larger prediction sets.
  • CPP augments base pruning importance with nonconformity-gradient sensitivity measuring first-order score displacement from setting a weight to zero.
  • Candidate-label variants are evaluated because true-label sensitivity is only a proxy for sets containing multiple plausible labels, with disjoint data roles protecting final calibration.
  • Threshold-aware CPP reaches 8.4 for CPP-Wanda and 7.8 for CPP-SparseGPT, but incurs explicit accuracy and gradient-compute costs.
  • Matched controls show that generic supervised gradients explain much of CPP’s improvement, so the evidence supports objective-specific efficiency effects rather than uniform superiority over gradient pruning.

2 Related Work

The paper situates CPP among pruning, calibration, conformal-prediction, and gradient-based retention methods. It differs by targeting post-training language-model pruning while keeping validation selection independent of final conformal calibration.

  • Existing pruning work includes magnitude, activation-aware Wanda, reconstruction-based SparseGPT, structured pruning, and gradient-based parameter-retention methods.
  • Calibration methods adjust confidence after training, while related pruning studies often report calibration as a byproduct rather than directly targeting conformal score geometry.
  • CPP’s closest direct predecessor applies inductive conformal prediction to pruning, whereas CPP targets post-training language-model pruning with independent validation selection and final conformal calibration.
  • Other related work optimizes conformal score functions, answer-choice sets, generation, factuality, or shift-aware guarantees under different assumptions.

3 The Reliability Contract Under Compression

The reliability contract ranks marginal coverage as the validity constraint, mean conformal set size as the primary efficiency outcome, and accuracy as a separate utility guardrail. The protocol enforces this hierarchy through fixed label scoring and disjoint data roles.

  • Decoder labels use fixed verbalizer sequences, with multi-token labels scored jointly rather than conformal prediction applied token by token.
  • Dprune constructs pruning statistics, validation subsets support provisional quantities and λ selection, and the independent conformal split is accessed only after model freezing.
  • The conformal quantile is the k-th smallest calibration score when k ≤ n and becomes +∞ when k > n.
  • Marginal coverage is the validity constraint, mean set size is the primary efficiency outcome, accuracy is task utility, and ECE is diagnostic.
  • Independent split conformal supplies validity for fixed θ′, while the validation guardrail limits accuracy loss.

4 Why Recalibration Alone Is Not Enough

Recalibration can restore marginal coverage after pruning but cannot recover lost label-specific separation, so valid prediction sets may remain inefficient. Corrected experiments compare pruners at matched empirical coverage while reporting accuracy separately.

  • Coverage is matched at 0.901–0.902 in the corrected Qwen2.5-1.5B/DBpedia-14 comparison, leaving set-size efficiency and accuracy as separate differences.
  • Top-1 accuracy depends on the largest class score, whereas set size depends on all candidate-label scores near the calibration threshold.
  • A scalar temperature can move confidence globally but cannot restore label-specific separation lost by the sparse model.
  • Reliability degradation can vary nonmonotonically with sparsity, so every reported sparsity is measured directly rather than interpolated.
  • The corrected analysis retires infeasible pilot pairs and uses only final-manifest results after checking the constraint that returning all K labels implies coverage one.

5 Calibration-Preserving Pruning

CPP modifies existing pruning importance scores with nonconformity-gradient sensitivity, using candidate-aware variants and disjoint data roles to target conformal set efficiency.

  • Method: CPP augments a base pruner’s coordinate importance with nonconformity-gradient sensitivity, rather than replacing the base pruning method.The base importance may come from magnitude pruning, Wanda, or SparseGPT reconstruction.
  • Method: True-label sensitivity is a diagonal empirical-Fisher-like factor for negative log-likelihood scores, but requires per-example gradients.Squaring a batch-mean gradient would introduce cross-example terms and would not estimate the intended quantity.
  • Implementation: The implementation prunes dense attention and MLP matrices while retaining normalization parameters, embeddings, tied output heads, positional mechanisms, and KV-cache state.The base pruner determines the allocation rule within eligible modules.
  • Candidate-aware variants: Candidate-aware variants include top-3, all-label, and threshold-aware constructions that more directly represent prediction-set inflation.Threshold-aware saliency focuses on candidate labels near a preliminary dense-model conformal boundary.

6 Theory

The theory separates generic split-conformal validity from CPP-specific efficiency: independent calibration preserves coverage, while bounded score movement controls quantile and set-size changes.

  • Coverage: Split-conformal coverage holds for any pruned model fixed before final calibration, provided calibration and test examples are exchangeable and independent of model construction.The result applies equally to Wanda, SparseGPT, random pruning, or other fixed models.
  • Coverage: With almost surely distinct scores and k ≤ n, the coverage bound is less than 1 − α + 1/(n + 1).The proof uses the uniform exchangeable rank of the test score among calibration and test scores.
  • Efficiency stability: Bounded score perturbations imply bounded changes in k-th order statistics and therefore support stability analysis for conformal quantiles and prediction sets.The order-statistic lemma gives |a_(k) − b_(k)| ≤ ϵ when every paired score changes by at most ϵ.
  • Limitations: The stability condition is sufficient rather than a certificate of small prediction sets, and the diagonal bound may fail to capture cross-coordinate cancellation or Hessian effects at high sparsity.Only labels near the conformal threshold can inflate the set under the stated uniform bound.
  • Efficiency stability: The first-order displacement proposition formalizes CPP’s saliency as a local proxy for score movement along the pruning path.It uses a masked linear term under a twice-differentiable score with bounded Hessian.

7 Experiments

Experiments evaluate CPP under disjoint pruning, validation, conformal-calibration, and test roles across Qwen tasks, sparsities, controls, transfer settings, and offline-cost measurements. CPP generally reduces valid prediction-set size, but gains depend on objective, candidate-label construction, accuracy trade-offs, and substantial offline computation.

  • Large-label results: At 50% sparsity, empirical coverage remains 0.900–0.902 while CPP-SparseGPT reduces set size by 10.4%–19.6% across large-label datasets.CPP-SparseGPT improves accuracy in all three reported large-label datasets; CPP-Wanda improves accuracy on Banking77 and CLINC150 but loses 1.5 points on DBpedia-14.
  • Complete Qwen grid: Across 15 dataset–sparsity cells, CPP-Wanda has smaller mean sets in 14/15 and higher accuracy in 12/15, while CPP-SparseGPT achieves 13/15 and 11/15, respectively.These are directions of five-seed means, not significance claims.
  • Matched controls: Matched controls show generic supervised gradients explain much of the efficiency gain, while true-label CPP-Wanda is not statistically resolved from Wanda⊕SNIP.Threshold-aware CPP-Wanda reaches set size 8.4 in the matched control but retains less accuracy than Wanda⊕SNIP.
  • Transfer and cost: Transfer diagnostics find smaller sets and higher accuracy for CPP-Magnitude on RoBERTa-base and smaller sets for CPP-SparseGPT on Llama-3-8B, while the study remains limited to classification.CPP adds no inference-time parameters or computation, but offline costs are 4.0× Wanda and 1.9× SparseGPT in the reported Qwen measurements.
  • Robustness: Independent Qwen DBpedia-14 redraws report set size 10.27 ± 0.41 for SparseGPT versus 8.79 ± 0.34 for CPP-SparseGPT, with matched coverage.Nested calibration- and pruning-size diagnostics show the same direction but are point-estimate sensitivity sweeps on one fixed partition.

8 Conclusion

CPP is an efficiency-oriented pruning objective applied under independently restored conformal validity. Results support smaller valid sets, especially on large-label classification, while matched controls and transfer diagnostics narrow the claim.

  • CPP changes retained weights to keep candidate-label scores more separated after valid recalibration, targeting smaller conformal prediction sets.
  • Coverage comes from split-conformal calibration, while CPP targets efficiency and accuracy remains a separate utility measure.
  • CPP can improve conformal efficiency, especially with threshold-aware saliency, but generic supervised gradients explain much of the gain.
  • The deployment workflow requires freezing the sparse model and preprocessing choices before fresh conformal calibration on an untouched split.
  • The five-seed, 15-cell Qwen grid supports the primary claim, while RoBERTa-base and Llama-3-8B provide transfer diagnostics.

Limitations

The paper’s evidence is scoped to reliability-sensitive classification and depends on strict split independence, labeled gradient data, and approximate perturbation theory. Reproducibility and deployment claims remain bounded by unbenchmarked systems and limited transfer diagnostics.

  • Claims cover reliability-sensitive classification rather than free-form generation, and the reported RoBERTa and Llama results are narrow diagnostics.
  • Validity requires exchangeability and strict independence of the final calibration split from pruning and selection decisions.
  • CPP requires labeled pruning data and per-example gradients, while candidate-aware variants increase offline computation.
  • The perturbation theory provides sufficient rather than tight conditions, with strong uniform bounds and possible Hessian dominance at high sparsity.
  • Exact reproducibility depends on releasing manifests, model and tokenizer versions, prompts, λ values, metrics, environment details, and scripts.

Ethical Considerations

Ethical and operational reliability requires auditing more than global marginal coverage. The paper’s implementation and optional analyses emphasize conservative calibration, explicit evaluation guardrails, and deployment limits.

  • Marginal coverage does not guarantee subgroup, conditional, or harm-weighted coverage, so deployment should audit relevant subgroups and realistic shifts.
  • Empirical test coverage can fall below 0.90 in finite samples even when population marginal coverage is valid.
  • Split-conformal validity follows from exchangeability after the fitted predictor is fixed independently of calibration data.
  • The optional Learn-Then-Test rule controls risk over a finite candidate grid, but Algorithm 1 uses deterministic guardrails instead.
  • The implementation separates pruning, validation, and final conformal data while supporting streamed per-module saliency computation.
  • Selection uses disjoint validation-calibration and validation-evaluation subsets, accuracy and coverage guardrails, and a λ = 0 fallback.

C.1 Corrected Small-Label Results

Corrected small-label and diagnostic results use final manifests and explicitly bounded protocols. Small label spaces limit set-size resolution, while the available Llama-3-8B experiment remains a scoped diagnostic rather than evidence of scaling or generative guarantees.

  • C.1 Corrected Small-Label Results: The corrected small-label results use final five-seed Qwen2.5-1.5B entries with empirical coverage and mean set size reported with 95% confidence intervals.
  • C.1 Corrected Small-Label Results: Small label spaces limit set-size resolution, so these tasks serve primarily as useful controls.
  • C.2 Large-Label Efficiency Accounting: The final 50% results express CPP changes relative to matched bases using relative set-size and absolute accuracy changes.
  • C.2 Large-Label Efficiency Accounting: Directional W/L comparisons cover all 15 Qwen set-size cells and do not assert statistical significance.
  • C.2 Large-Label Efficiency Accounting: Figures 2–5 report final-manifest means, matching set-size views, large-label reductions, and 15-cell directional rates.
  • C.2 Large-Label Efficiency Accounting: The Llama-3-8B table reports perplexity, accuracy, coverage, and set size over three seeds at 50% sparsity.
  • C.2 Large-Label Efficiency Accounting: The Llama-3-8B experiment does not establish a scaling law or a guarantee for generative tasks.
Loading 2608.23744v1…