PaperlayerView exact revision on arXiv

Source-linked AI summary

Capability Scaling-Down Laws for LLM Compression

Xueqi Cheng, Liang Wu, Kelly Wan, Liangjie Hong, Yushun Dong

arXiv:2610.02462v1cs.LGcs.CL

TL;DR

Compression method selection remains empirical because similar resource reductions can produce different capability losses. This paper develops and evaluates predictive relations across pruning, quantization, and distillation, finding that shared pruning responses reduce measurement needs and that distillation outcomes depend on data reuse and evaluation distribution.

  • Problem

    Compression methods and configurations remain largely empirical because comparable resource reductions can produce substantially different capability losses, while prediction needs and transfer remain incompletely understood.

    Source: §1 · pp. 1–2

  • Method

    The paper studies a common framework for pruning, quantization, and distillation, testing predictive relations for accuracy, measurement efficiency, transfer, and selection value.

    Source: §1 · p. 2

  • Results

    Sharing the pruning density response halves configuration measurements and matches full-grid regression within 0.020 nats per token on mathematics and code, while fixed method priority attains numerical selection’s regret on question answering.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings clarify what information capability scaling-down laws need and provide an empirical basis for compression choices aligned with application-specific capability requirements.

    Source: §8 · p. 10

    Distillation responses are measured under one fixed recipe, and changing learning rate shifts mathematics and code loss increases by an order of magnitude.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

LLM compression reduces inference costs and memory requirements, but selecting a method and configuration remains largely empirical because comparable resource reductions can produce different capability losses. We systematically investigate capability scaling-down laws for LLM compression across pruning, quantization, and distillation. Our framework measures capability loss in mathematics, code generation, and question answering, and relates these measurements to model size, training stage, compression settings, data availability, and training exposure. We develop simple predictive relations and evaluate their accuracy, measurement efficiency, and generalization to unseen configurations and model states. Sharing the density response across pruning levels halves the configuration measurements needed to fit a pruning predictor: on new Pythia states, on pre-registered OLMo-2 test states and under Wanda pruning, the compact relation matches a regression fitted with all measurements on math and code to within 0.020 nats per token, with coefficients refitted for each setting. Controlled distillation experiments show that the cost of heavy data reuse recurs across question-answering distributions, while the net benefit depends on the evaluation distribution. We further evaluate the decision value of these predictions by comparing numerical selection with configuration medians and fixed method priorities. Independent evaluations across two model families show that selection captures most of the available cross-method benefit for question answering within the tested candidate sets, where a fixed method priority attains the same regret, with smaller opportunities for mathematics and code. These results clarify the predictive scope of capability scaling-down laws and their use in compression method selection. Our code is publicly available at: https://github.com/LabRAI/scaling_down_law.

1 INTRODUCTION

Compression can cut inference costs and memory use, but comparable resource reductions can produce different capability losses, leaving method and configuration selection largely empirical. The paper studies predictive relations across compression methods and evaluates their measurement efficiency, transfer, and value for selection.

1 INTRODUCTION

  • Compression selection remains largely empirical because comparable resource reductions can produce substantially different capability losses.Repeatedly compressing and evaluating every candidate can be needed to identify configurations that meet workload-specific capability requirements.Source: §1 · p. 1
  • Existing predictions provide limited evidence on measurement needs, transfer across model families or algorithms, and when numerical forecasts change method choice.A configuration can preserve overall performance while degrading a capability required by a workload.Source: §1 · p. 2
  • The paper systematically investigates capability scaling-down laws for pruning, quantization, and distillation using a common framework.It examines response sharing and dependence on model state, training conditions, and evaluation distribution.Source: §1 · p. 2
  • Sharing the pruning density response halves configuration measurements and matches full-grid regression within 0.020 nats per token on math and code.The result reproduces on another model family and pruning criterion, with coefficients estimated anew for each setting.Source: §1 · p. 2
  • Independent selection tests find that a stable method priority attains numerical selection’s regret on question answering, with smaller additional value on mathematics.The tests distinguish the value of comparing methods from the extra value of numerical prediction.Source: §1 · p. 2

2 PROBLEM FORMULATION AND EXPERIMENTAL DESIGN

The study measures within-model changes in reference-completion loss across capabilities and controlled compression interventions. Its design varies model state and intervention settings, then evaluates prespecified predictors on unseen settings and states.

2.1 CAPABILITY LOSS AND PREDICTION INPUTS

  • Capability loss is token-weighted conditional cross-entropy on fixed probes for mathematics, code generation, and multi-hop question answering.The probes come from held-out MATH-500, MBPP, and 2WikiMultihopQA splits.Source: §2.1 · p. 2
  • Pruning and quantization use within-model loss change from the pre-intervention checkpoint, while distillation uses signed transfer change from the student’s initial state.Absolute loss levels across different tokenizers are not placed on one scale; task accuracy and execution success are evaluated separately.Source: §2.1 · p. 2

2.2 MODELS AND COMPRESSION METHODS

  • The controlled implementations expose weight density, bit-width, group size, training data, and exposure as intervention variables.Pythia varies model size and training stage separately, while Gemma students support comparisons of data availability and reuse.Source: §2.2 · pp. 2–3
  • The protocol uses global magnitude pruning, per-output-channel or grouped round-to-nearest quantization, and LoRA fine-tuning on teacher traces.Wanda is included as a second pruning criterion.Source: §2.2 · p. 3
  • Pythia development states span three sizes and training stages; held-out Pythia states and OLMo-2 checkpoints test transfer.A twelve-model panel from 0.6B to 32B establishes the response phenomena.Source: §2.2 · p. 3
  • Predictors may use parameter count, disclosed pretraining-token count, and reference capability losses, with method settings supplying the intervention inputs.For distillation, the initial student is the reference state.Source: §2.2 · p. 3

2.3 FITTING AND EVALUATION

  • Predictors are fitted on development states and settings, fixed before evaluation, and tested on unseen settings and states against same-input alternatives and source-free references.Error is mean absolute error in predicted loss change on matched cells, with the strongest baseline selected inside development folds.Source: §2.3 · p. 3

3 EMPIRICAL ANALYSIS OF CAPABILITY LOSS

Compression responses differ across capabilities, models, and intervention axes. Controlled experiments identify shared pruning patterns for mathematics and code, configuration-dependent quantization responses, and distillation effects that vary with data reuse and evaluation distribution.

3.1 RESPONSES TO PRUNING AND QUANTIZATION

  • Pruning losses rise steeply below density 0.8; quantization losses are large at 3 bits and small from 4 bits, while grouped 3-bit loss grows with group size.At 4 and 5 bits, the grouped response stays near zero; the bit-width threshold depends on group size.Source: §3.1 · p. 3
  • Across twelve models, capability ordering changes between methods and same-setting loss magnitudes differ by two orders of magnitude.On the controlled Pythia panel, mathematics and code share a density-response shape with state-dependent scale, while question answering has no shared curve.Source: §3.1 · p. 3, Figure 1 · p. 4, Figure 2 · p. 4
  • Using parameter count, pretraining tokens, and reference loss reduces mean absolute error from 0.41 to 0.25 nats for mathematics and from 0.52 to 0.29 for code, but raises it from 0.50 to 0.81 for question answering.The mathematics and code pruning amplitude depends on model state, whereas question answering does not follow a shared proportional density curve.Source: §3.1 · pp. 3–4

3.2 EFFECTS OF TRAINING DATA AND EVALUATION DISTRIBUTION

  • At reuse ratio near twelve and a matched budget near 200k tokens, question-answering loss rises by +0.29, +2.55, and +4.85 nats on 100M, 700M, and 3.2B students.Nine times more independent data turns the same budget into a gain near one nat on each; all three evaluated distributions worsen at high reuse and improve as reuse falls, but only fresh 2Wiki crosses into a gain.Source: §3.2 · p. 4, Figure 3 · p. 5
  • For the fresh 2Wiki sample, measured independent-data intervals meeting a loss constraint lie within tested pools for both 1B and 4B students at the two larger budgets.This was a prespecified experiment of eighteen trajectories.Source: §3.2 · p. 5
  • The distillation responses are recipe-dependent: doubling the learning rate from 5 × 10−5 to 1 × 10−4 raises mathematics and code loss increases by an order of magnitude.The reported responses otherwise use one fixed recipe; at the lower rate, the 1B student is almost unharmed on mathematics and code, and question answering improves by 0.79 nats.Source: §3.2 · p. 5

4 CAPABILITY SCALING-DOWN LAWS

The predictive relations follow the observed response structure: pruning shares a density shape across states, quantization depends on the full configuration, and distillation separates data availability, exposure, and evaluation distribution. Their forms and transfer scope differ by method.

4 CAPABILITY SCALING-DOWN LAWS

  • Predictors use the observed response structure, method inputs, and a validation target; initial-state information enters only where it lowers development error.Pruning shares a density shape, quantization uses the full configuration, and distillation separates data, exposure, and evaluation distribution.Source: §4 · p. 5

4.1 PRUNING

  • For mathematics and code pruning, a state-dependent amplitude multiplies a density term shared across states.The five-parameter-per-capability form uses standardized inputs; sharing the density term lets each state retain representation while measuring fewer densities.Source: §4.1 · p. 5

4.2 QUANTIZATION

  • Quantization predictors follow measured configurations: grouped quantization interpolates linearly in log2 group size for seen states and otherwise uses the per-configuration development median.Per-channel quantization uses per-bit regression on fitted states and the per-bit median on new states; validation covers unseen group sizes and a new state.Source: §4.2 · p. 6

4.3 DISTILLATION

  • Distillation predictors use the reuse ratio for mathematics and code, and a joint budget-and-pool form for question answering, as selected by leave-one-run-out comparison.The relations describe Gemma-3 270M and 1B students under one recipe, with pools up to 158k tokens, budgets of 50k–200k, and QA scored on 2Wiki; the 4B student and reuse near fourteen are outside this domain.Source: §4.3 · p. 6

5 PREDICTIVE PERFORMANCE AND GENERALIZATION

Compact pruning prediction reuses a shared density response to reduce measurements, while prediction quality and transfer depend on whether configurations or model states are unseen. Results also identify task- and setting-specific limits to generalization.

5.1 MEASUREMENT EFFICIENCY

  • The measurement saving is in density measurements, not model states; at 9 measurements the per-density regression cannot be fitted, and the reported errors do not test statistical equivalence.The compact form’s median error changes little between 18 and 36 measurements, after falling between 9 and 18.Source: §5.1 · p. 6
  • Within 0.020 nats per token of full-grid errors on math and code, the half-grid compact pruning form uses 18 of 36 Pythia or 12 of 24 OLMo-2 configurations.The comparison covers new Pythia states, OLMo-2 test states, and Wanda pruning; the reduced grids retain every development state.Source: §5.1 · p. 6, Table 1 · p. 7, Figure 4 · p. 7
  • Without refitting, Pythia coefficients err by 0.4 to 1.9 nats on OLMo-2, while magnitude-pruning coefficients err by 0.19 to 0.27 nats under Wanda.The predictive form and fitting strategy transfer, but coefficients are estimated for each setting.Source: §5.1 · p. 7

5.2 PREDICTION AT UNSEEN CONFIGURATIONS AND MODEL STATES

  • For unseen configurations of seen states, relations reading the initial state help; for new model states, source-free medians predict as well or better.At equal development budget, predictions were fixed before measurement.Source: §5.2 · p. 7, Table 2 · p. 7
  • At six new distillation pools and three new budgets, every selected form improves on zero change; for new larger states, median curves outperform source regressions on pruning and per-bit quantization.At 6.9B and step 112k, median versus source-regression errors are 0.05 versus 0.23 for pruning, 0.07 versus 0.19 for per-bit quantization, and 0.15 versus 0.96 for grouped quantization; outside the tested density range, source-conditioned forms err by 0.6 to 4.5 nats.Source: §5.2 · p. 8

6 COMPRESSION METHOD SELECTION

The selection rule compares predicted capability losses under nominal storage constraints. On the tested candidates, it captures most of the question-answering opportunity over quantization, while simpler medians or fixed priorities can attain the same regret.

6 COMPRESSION METHOD SELECTION

  • On 64 feasible question-answering cells, the final rule selects the new student in 63, with regret of 0.000 nats on three references and 0.027 on the fourth, versus 0.41 to 1.84 for quantization only.The four references supplied no outcomes to the rule; the comparison spans twelve candidates and seventeen storage budgets per reference.Source: §6 · p. 8
  • On 384 fresh question-answering items, the rule’s choice has lower loss than quantization only on three references and is level on the fourth, but loss and exact-match rankings can differ.A generation readout puts the same students ahead on exact match on three references, behind on one, and no better than their initial students; the paper distinguishes termination from evaluation target.Source: §6 · p. 8
  • On Pythia 410M, quantization-only regret is 0.59 nats, decomposed into 0.47 selection error within quantization and 0.12 cross-method opportunity.The rule’s choice is 0.09 nats better than the best quantization candidate.Source: §6 · p. 9
  • On question answering, fixed method priority, development medians, and the final rule reach the same regret: 0.014 nats on the Pythia pair and 0.000 on the Gemma pair.The regret is 0.58 and 1.12 for quantization only, respectively; the benefit rests on a stable difference between methods rather than fine loss margins.Source: §6 · p. 9
  • For mathematics, medians beat fixed priority by 0.024 nats on the Gemma pair, while the parametric relations improve on neither selection baseline.The medians use only the development panel, so this benefit requires no measurement of the target model.Source: §6 · p. 9

7 RELATED WORK

Prior work develops compression laws and documents uneven capability losses, but the paper focuses on measurement needs, transfer, and the decision value of prediction across methods.

7 RELATED WORK

  • Earlier compression laws predict aggregate loss from compression inputs, while task-specific pruning and quantization laws address capability responses and unseen configurations (Sengupta et al., 2026; Zhou et al., 2026).The paper adds analysis of measurement needs, transfer across model families and pruning criteria, and decision value.Source: §7 · p. 9
  • Prior comparisons report method-dependent capability retention and evaluation-sensitive efficiency, but do not specify which method a model, capability, and budget should receive without measuring candidates (Jaiswal et al., 2024; Panferov et al., 2025; Zhang et al., 2026).The paper tests that selection decision on independent candidates.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents a systematic study of capability scaling-down laws across compression settings. Its predictive relations and independently evaluated selection method provide an empirical basis for relating compression outcomes to capability requirements.

8 CONCLUSION

  • The study characterizes how capability losses vary with model state, compression configuration, and training data, identifying shared response patterns and capability differences.The framework covers pruning, quantization, and distillation.Source: §8 · pp. 9–10
  • Compact predictive forms reduce measurement requirements while maintaining useful accuracy, and the paper independently evaluates method selection under nominal storage constraints.The conclusion frames these contributions as clarifying the information needed to predict compression outcomes and align choices with application-specific capability requirements.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

The repository provides the scripts and recorded results used to generate the paper’s numbers, tables, and figures.

REPRODUCIBILITY STATEMENT

  • The public repository contains the measurement suite, recorded results, table and figure generators, and tests for every printed value.Section 2 and Appendices A and I describe the protocols and when predictions were fixed.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The appendix documents the study’s protocols, probes, model cohorts, and measurement checks. It reports that QA loss and behavior do not always move together, and that results depend on the reference condition.

ETHICS STATEMENT

  • The study uses public models and benchmark data, involves no human subjects or personal data, and reports no foreseen direct negative use.The appendix records experimental settings, fitted models, prediction results, mechanism analyses, and selection evidence.Source: ETHICS STATEMENT · p. 10, APPENDIX · p. 15

A.1 PROBES AND LOSS COMPUTATION

  • Capability losses are measured on 64 held-out probes per capability, separated from the estimation probes used for gradient and Fisher statistics.The primary probes come from MATH-500, MBPP, and 2WikiMultihopQA; the study also specifies secondary benchmarks for measurement checks.Source: §A.1 · p. 15

A.2 COMPRESSION PROTOCOLS

  • The compression protocols include per-channel and grouped quantization, with the latter sharing a scale across groups of weights.The quantization loss shape is not derived from squared step-size ratios, and the study reports nominal storage ratios rather than measured memory or latency.Source: §A.2 · p. 15

A.3 THE CONTROLLED PYTHIA PANEL

  • The Pythia panel varies model size and training stage, and prospective test sources were excluded from fitting and model selection.The appendix also documents checkpoint identity checks and the exclusion of repeated 2.8B weights.Source: §A.3 · p. 17

B MEASUREMENT VALIDITY

  • A lower QA loss does not necessarily mean better QA behavior: four pruned OLMo-3-32B cells below dense loss show no increase in exact match.The loss-to-behavior audit covers pruning only; the smaller public panels for quantization and distillation were not audited in the same way.Source: B MEASUREMENT VALIDITY · p. 17
  • The reported loss measurements use the full-solution reference; alternative reference conditions differ for math and QA but coincide for code.The results should not be read as evidence of robustness to reference format.Source: B MEASUREMENT VALIDITY · p. 17
  • Within-model responses are differences in native-token nats per token, while absolute cross-family levels are normalized per byte or reported separately.Negative responses are retained rather than clipped at zero.Source: B MEASUREMENT VALIDITY · p. 18

C HETEROGENEOUS-MODEL RESULTS

Compression responses vary across models and capabilities, and pruning-response shape transfers across criteria more readily than its scale. Quantization effects depend strongly on bit width, while pruning robustness is not monotonic in model size.

C HETEROGENEOUS-MODEL RESULTS

  • At the same pruning density, OLMo-3-32B loses 0.01 nats on math while Gemma-3-27B loses 4.0, and QA’s relative damage varies by model series.For Qwen3 and OLMo-3-32B, QA is least damaged before the math cliff; in Gemma-3 sizes, Gemma-4-31B, and Muse-30B, it is most damaged at the mildest densities.Source: C HETEROGENEOUS-MODEL RESULTS · pp. 18–19, Table 4 · p. 19
  • Int8 and int6 remain within 0.01 nats of dense, while int3 collapses 12 of 14 models by 4.6 to 28 nats, with OLMo-3 less affected.For held-out bit widths, the learned exponential fits better than the fixed 4−b shape, at 6.7 versus 7.6 pooled nats.Source: C HETEROGENEOUS-MODEL RESULTS · pp. 19–20, Table 5 · p. 20
  • With the power-form exponent pooled from magnitude-pruning cells and only scale refit, Wanda responses are matched within 0.15 and 0.08 nats on math, and 0.23 and 0.29 on code.Wanda changes response amplitude by an order of magnitude; its shape transfers better than its scale, which must be measured under the new criterion.Source: C HETEROGENEOUS-MODEL RESULTS · p. 19, Table 6 · p. 20
  • In paired Gemma-3 distillation experiments, answer-only training raises math loss about twice as much as full-trace training, while QA improves by 0.6 to 1.4 nats under both recipes.Because the recipes differ in supervised tokens and update counts, the pairing does not isolate reasoning coverage; controlled comparisons address this under a controlled recipe.Source: C HETEROGENEOUS-MODEL RESULTS · pp. 20–21

C.1 SCALE AND FAMILY STRUCTURE

  • Gemma-3 robustness peaks at 12B and 27B collapses earlier; across families, pruning loss at d = 0.7 ranges from 0.02 nats on OLMo-3-32B to 4.04 on Gemma-3-27B.The proposed gradient-concentration statistic was retired because its robustness ordering inverted across families and failed within Gemma-3.Source: §C.1 · p. 21

E.1 SELECTION AND BOUNDARY OF THE DELIVERED RELATIONS

The section selects and tests compression-specific predictive relations, then establishes where sharing response structure across capabilities and settings is useful. Its boundaries vary by method, model state, capability, and evaluation range.

E.1 SELECTION AND BOUNDARY OF THE DELIVERED RELATIONS

  • On three new checkpoints, the five-parameter power form matches A2's math error but does not beat the median on code or QA; on a 2.8B source, the median is 3–7 times better.The development comparison found A2 and the power form within 0.02 nats, at 0.479 and 0.494, respectively; the power form was preferred for vanishing at d = 1.Source: §E.1 · p. 22
  • On the new quantization state, the source-free median is as good or better except for code; interpolation holds at unseen group sizes of seen states, and the surface is not delivered.In the confirmation round, interpolation won the pre-specified math and code cells, while the median won on the new state.Source: §E.1 · p. 22
  • Distillation's additive joint structure is not established; at new pools, the selected form beats the strongest baseline for code on both students, QA only on 270M, and loses to intercept baselines on math.The tested student range does not include a student 6.6 times the reference size, and the new-pool result does not establish a full student-data-training law.Source: §E.1 · p. 23

E.2 APPLICABILITY ACROSS CAPABILITIES

  • Capability conditioning helps when response shapes differ: QA's pruning response changes sign across density, while math and code are as well or better served by a shared curve.The QA aggregate curve also varies in direction across sources; shared curves are indistinguishable for grouped quantization, and the distillation comparison is algebraically equivalent.Source: §E.2 · p. 23

E.3 SHARED STRUCTURE, PARAMETER RANGES, AND ONE-POINT CALIBRATION

  • A single function family holds across methods only for pruning; shared exponents cost little there, quantization coefficients must be capability-specific, and leave-one-source prediction intervals under-cover.The pruning intervals cover 67–78% of held-out cells at a mean full width near 1.5 nats; one-point calibration offers no saving over the uncalibrated predictor.Source: §E.3 · p. 23, §E.2 · p. 23
  • Separate capability responses improve pruning error by 0.059 nats macro-averaged (0.168 for QA), while the gain is 0.003 for quantization and zero for algebraically identical distillation forms.The QA gain reflects a sign reversal across density that no proportional shared-curve baseline can represent; the ablation is retrospective.Source: §E.3 · pp. 23–24

F PRE-SPECIFIED PREDICTION RESULTS

Pre-specified tests assess prediction transfer, measurement efficiency, and data requirements across compression methods and model states. The compact pruning relation retains accuracy with fewer configuration measurements, but its coefficients remain setting-specific and the distillation data-requirement relation does not outperform monotone interpolation.

F PRE-SPECIFIED PREDICTION RESULTS

  • The QA corner tests do not distinguish additive from interaction structure, while math and code were pre-specified as underpowered; on distillation, post-hoc source-conditioned forms have the lowest error for each student and capability.Distillation confirmation used six new pools at U=200 and three token budgets; pool-cluster intervals cover pool sampling, not item-level uncertainty.Source: Figure 9 · p. 25, Figure 10 · p. 26, F PRE-SPECIFIED PREDICTION RESULTS · p. 27
  • At 6.9B, the source regression errors exceed the median curve on math, code, and QA; at density 0.55, source-free curves win under both protocols, while quantization performance varies by protocol and bit width.At 112k under protocol A, the source regression gives 0.23, 0.19, and 0.96 nats versus 0.05, 0.07, and 0.15 for the median on math, code, and QA; at 3 bits it overshoots by about four nats.Source: F PRE-SPECIFIED PREDICTION RESULTS · pp. 25–26

F.1 MEASUREMENT EFFICIENCY OF THE COMPACT PRUNING FORM

  • On four new Pythia states, the reduced-budget power form meets the pre-specified within-0.05-nats criterion on math and code, while the median curve is the better QA predictor at every development budget.At 18 measurements, the power form's median errors are 0.20 nats on math and 0.17 on code, versus 0.18 and 0.13 for the full-grid regression at 36 measurements.Source: §F.1 · p. 27, Figure 12 · p. 28
  • On OLMo-2, the reduced-grid compact form errs by 0.017 nats on math and 0.020 on code, versus 0.019 and 0.027 for full-grid regression; on new items the corresponding pairs are 0.012/0.013 and 0.029/0.036.The reduced grid uses 12 rather than 24 configurations and 59% of full-grid GPU seconds; Pythia coefficients applied without refitting do not transfer.Source: §F.1 · pp. 28–29, Table 14 · p. 30
  • Under Wanda, the reduced-grid compact form is within 0.020 nats of full-grid regression on math and code; its errors are 0.023 and 0.026 versus 0.025 and 0.027, respectively.The relation's structure and measurement saving replicate under this second pruning criterion, but its coefficients are criterion-specific; the reduced grid costs 70% of full-grid GPU seconds.Source: §F.1 · pp. 29–30, Table 15 · p. 31

F.2 MEASURED DATA REQUIREMENTS UNDER DISTILLATION

  • For measured distillation data requirements, the pre-specified boundary relation did not outperform per-student monotone interpolation, which solved the most requests on QA, math, and code distributions.The experiment used 1B and 4B students, three pool tiers, three seeds, and budgets of 50k, 100k, and 200k supervised tokens.Source: §F.2 · p. 31

G MECHANISMS

  • The mechanism analyses examine pruning response in deleted capability mass, the first-order term's sign, cliff geometry, and logit displacement as an account of loss change.Source: G MECHANISMS · p. 31

F.1 MEASUREMENT EFFICIENCY OF THE COMPACT PRUNING FORM

  • For task accuracy, the pruning-law form predicts better than the loss route: mean absolute errors are 0.145 versus 0.264 without calibration and 0.119 versus 0.312 with one measurement.The endpoint comparison does not replace the paper's capability-loss claims; the loss route can be limited by accuracy-floor effects at 64 probes.Source: §F.1 · pp. 30–31, Table 17 · p. 32

G.1 THE PRUNING LAW

The pruning law models pre-cliff loss through a signed first-order alignment term and a power-law curvature residual in deleted capability mass. Further analyses connect the signed response to capability-specific improvements, locate pruning cliffs by a family-specific geometric threshold, and test a second-order account of compression loss changes.

G.1 THE PRUNING LAW

  • 2.17 nats MAE for the shared-η mechanism beats 10.3 for per-capability raw sparsity and 8.2 for direct damage in leave-one-family-out transfer.Its edge over raw sparsity within-model is not statistically significant; without the first-order term, QA is unfittable.Source: §G.1 · p. 32
  • In deleted-mass coordinates, η concentrates at 1.2–2.2 across families, and one shared η per model carries capability differences through two measured quantities.The mechanism uses dense-model gradient information, so it is an upper comparator rather than a predictor from the initial state.Source: §G.1 · p. 33

G.2 THE SIGNED FIRST-ORDER TERM PREDICTS, IN BOTH DIRECTIONS

  • 67.6% sign prediction accuracy across 108 mild-regime cells; beyond that regime, first- plus second-order predictions fail by orders of magnitude at the cliff.The family-level pattern includes Qwen and OLMo-3-32B QA improvements, no Gemma QA improvement, and an OLMo size flip.Source: §G.2 · pp. 33–34

G.3 THE CLIFF IS A GEOMETRIC THRESHOLD CROSSING

  • Capability-region retention crosses family-specific cliff thresholds—about 0.04–0.10 for Gemma and 0.13–0.23 for Qwen—consistent across sizes and capabilities within each family.The evidence is preliminary: three models across two families.Source: §G.3 · p. 34

G.4 THE LOGIT-DISPLACEMENT ACCOUNT OF THE COMPRESSION LOSS CHANGE

  • Median relative error for the second-order displacement account stays below 5% when loss change is below about 0.1 nats, then rises above 20% by 0.25 nats.The account explains measured loss changes after compression; its residual variance requires the compressed model and is not a pre-compression predictor.Source: §G.4 · p. 35, Figure 15 · p. 35, Table 19 · p. 35

H.1 INDEPENDENT CROSS-METHOD VALIDATION

The authors validate a pre-specified compression-selection rule on independent states and compare it with retrospective alternatives. Question answering was the only objective with meaningful cross-method opportunity in that confirmation, while generation readouts showed that loss-based selection does not determine exact-match outcomes in every setting.

H.1 INDEPENDENT CROSS-METHOD VALIDATION

  • Question answering was the only objective with an opportunity to capture, and the rule captured it; opportunity was negligible for the other three objectives across the independent validation.All four references met the pre-specified criterion; math, code, and QA were confirmed, while the multi-capability objective was prospective and unconfirmed.Source: §H.1 · pp. 36, 42, Table 21 · p. 37
  • On the independent validation, a fixed method priority matched the rule where opportunity lay within one method, while development medians added value for Gemma 3 1B mathematics.The numeric value of prediction was carried by development medians already used by the rule; source-conditioned relations contributed measurement efficiency instead.Source: §H.1 · p. 37, Table 23 · p. 39

H.2 PROTOCOL SENSITIVITY OF THE GENERATION READOUT

  • On fresh QA items, distilled students had lower completion loss than quantization-only choices on three references, but 96-token exact match favored them on three and trailed on Pythia 1.4B.The loss ordering and generation ordering can diverge; the selection rule optimizes reference-completion loss, not a deployment-specific answer-format or accuracy endpoint.Source: §H.1 · pp. 37–38

H.1 INDEPENDENT CROSS-METHOD VALIDATION

  • Byte-based feasibility preserved the within-family selection picture, but changed feasible sets in 176 of 272 cells and moved the rule’s choice outside the feasible set in 10.Sparse-weight indices and student embeddings account for important differences from nominal storage ratios.Source: §H.1 · pp. 38–39
  • On 17 controlled states, the law-based choice was within 0.19, 0.17, and 0.42 nats of the oracle for math, code, and QA, respectively, and chose its method in 96%, 95%, and 73% of 289 cells.Quantization-only was close behind.Source: §H.1 · p. 39

H.2 PROTOCOL SENSITIVITY OF THE GENERATION READOUT

  • Adding supervised end-of-sequence markers reduced cap-reaching generations from 0.63 and 0.99 to 0.01 and 0.00 for the Pythia and Gemma students, respectively.The marker left reference-completion loss unchanged; generation accuracy improved for Gemma but not Pythia.Source: §H.2 · pp. 42–44

I REPRODUCIBILITY AND TIMING OF PREDICTIONS

The section specifies how prediction timing was protected and distills four lessons about fitting, transferring, and identifying compression laws. It also documents fresh-state selection maps and appendix controls.

I REPRODUCIBILITY AND TIMING OF PREDICTIONS

  • Figure 18 maps predicted methods against nominal storage budget on four fresh states, marking oracle disagreements and cases with no clear winner.Its last panel uses the largest loss increase across capabilities as the objective.Source: Figure 18 · p. 45
  • The appendix captions specify an end-marker control on 384 fresh QA items and an information-budget comparison of zero versus one calibration measurement, with an oracle diagnostic.The end-marker control compares original students with students retrained using appended and supervised end-of-sequence tokens.Source: Table 28 · p. 45, Table 29 · p. 45
  • Prospective tests fixed forms, development data, standardization, and coefficients before measuring a new source’s reference losses and recording candidate predictions ahead of outcomes.The prediction forms and measurement-efficiency confirmation were specified before the relevant models were trained or pruned.Source: I REPRODUCIBILITY AND TIMING OF… · pp. 44, 46

J DISCUSSION

  • Four lessons distinguish rejected functional forms from whole model classes, predictors from explanations, source fit from transfer, and identifiability from experimental design.The discussion also limits each relation to its evaluated distributions, implementations, and configuration ranges.Source: J DISCUSSION · p. 46

Related papers

Loading 2610.02462v1…