How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

1 Introduction

• The authors frame the work as an investigation of internal causes of degradation, rather than a new compression algorithm or another account based only on pruned-parameter importance.Source: §1 · p. 1
• The study uses parameter pruning to examine whether changes in explicit inference patterns explain why pruning different parameters leads to diverse performance degradation.Source: §1 · p. 1
• Interactions are nonlinear relationships among input variables encoded by a DNN, providing a representation for examining its inference logic.Source: §1 · p. 1
• Pruning has a distinct three-phase interaction dynamic: performance remains largely unaffected until pruning begins removing low-order interactions, whose removal accompanies significant degradation.The figure also describes test accuracy as not decreasing noticeably until generalizable interactions are removed.Source: Figure 1 · p. 2, §1 · p. 2
• Sensitive-module pruning significantly removes low-order interactions, whereas pruning insensitive modules removes only high-order interactions.The latter modules do not significantly degrade DNN performance.Source: §1 · p. 2
• Low-order interactions generalize more strongly; about half of high-order interactions have positive effects and half negative, so their effects offset and act like noise patterns.Source: §1 · p. 2

2.1 Preliminaries: interactions

• The method uses a logical surrogate composed of AND and OR interactions to approximate a DNN’s output across masked input states.For image classification, a variable is an image patch; for language modeling, it is one or more tokens corresponding to a word.Source: §2.1 · pp. 2–3

2.2 Quantifying the changes in interactions during progressive parameter pruning

• The method classifies interactions as preserved, removed, or emerged after pruning, and measures preserved low- and high-order ratios and the offsetting ratio κ of removed effects.Removed high-order interactions often have a spindle-shaped distribution: half produce positive effects and half negative effects, so they greatly offset each other.Source: §2.2 · pp. 4–6

2.3 Three-phase dynamics of interactions during progressive pruning

• Progressive pruning shows three phases: high-order interaction removal initially leaves accuracy unchanged, low-order removal accompanies sharp degradation, and excessive pruning leaves accuracy very low.Phase 1: below 14.6% pruning for ResNet-56 and 59.8% for VGG-19, accuracy remains around 92.1%, with ρhigh = 0.3 ± 0.1, ρlow = 0.82 ± 0.03, and κ = 0.96 ± 0.02. Phase 2: at 39.7% and 77.4%, respectively, accuracy drops to around 10.3%, with ρhigh = 0.01 ± 0.01, ρlow = 0.1 ± 0.1, and κ = 0.75±0.05; Phase 3: at 50.2% and 99.2%, accuracy remains around 10.0%, with ρhigh = 0.01 ± 0.01 and ρlow = 0.1 ± 0.1.Source: §2.3 · p. 7

2.4 Specificity of removed interactions under generic pruning settings

• Pruning sensitive modules significantly removes low-order interactions, and the removed interactions have poorer offsetting ratios.The study compares module-level pruning using preserved low/high-order interaction ratios and the offsetting ratio of removed interactions.Source: Figure 5 · p. 8
• Insensitive-module pruning preserves ρlow = 0.7 ± 0.1 with κ = 0.92 ± 0.03, while sensitive-module pruning preserves ρlow = 0.4 ± 0.2 with κ = 0.8 ± 0.1.Insensitive pruning mainly removes high-order interactions; sensitive pruning removes both low- and high-order interactions, and non-offsetting low-order removals are identified as the direct cause of degradation.Source: §2.4 · pp. 8–9

2.5 How do removed interactions correlate with DNN generalization degradation?

• Generalizable interactions are mainly concentrated at low orders, and their removal is identified as a primary factor in performance degradation.An interaction is defined as generalizable when a reference DNN trained on unseen samples also encodes it with a consistent effect.Source: §2.5 · pp. 9–10, Figure 6 · p. 9
• Sensitive-module pruning removes ρremoved = 0.35 ± 0.05 of generalizable interactions, compared with 0.1 ± 0.02 for insensitive-module pruning.Figure 7 (left) presents the removed-generalizable-interaction ratios for the two module types.Source: §2.5 · p. 10, Figure 7 · p. 10
• At low pruning ratios, performance and generalizable interactions remain unaffected; at higher ratios, both degrade as generalizable interactions are removed.The passage associates low pruning ratios with Phase 1 and high ratios with Phases 2 and 3.Source: §2.5 · p. 10, Figure 7 · p. 10

3 Conclusion

• Across twelve DNNs, including four LLMs and eight image classifiers, degradation is largely attributed to removing generalizable low-order interactions; a rigorous mathematical proof remains lacking.Source: §3 · p. 10
• Preliminary follow-up work reports that co-training DNNs while penalizing non-transferable interactions improves interaction generalizability and helps maintain performance at higher pruning ratios.Source: §3 · p. 10

A Related work on interaction-based explanations

• Prior work quantified input-variable interactions [20], linked smooth DNN outputs to few interactions, extracted commonly encoded interactions [4], and optimized extraction baselines.Source: A Related work on interaction-based… · p. 15
• Prior studies associated higher-order interactions with poorer test generalization [6], adversarial robustness, and mean-field Bayesian modeling [24].Source: A Related work on interaction-based… · p. 15
• The paper addresses a gap in post-training interventions by connecting interaction-based explanations with parameter pruning to study changes in representation quality and generalization.Source: A Related work on interaction-based… · p. 15
• Mechanistic interpretability studies component roles, whereas interaction-based explanations treat the network as a whole and represent its inference logic with AND-OR interactions [26] [27].Source: A Related work on interaction-based… · p. 15

B Related work on parameter pruning

• Pruning research spans weight pruning, quantization, and coding, one-shot methods such as SparseGPT [16] and Wanda [17], and structural methods [2] [18].Source: B Related work on parameter pruning · p. 15
• Rather than propose another compression algorithm, this work investigates internal interactions as mechanisms underlying pruning-induced performance degradation, beyond explanations centered on pruned-parameter importance.Source: B Related work on parameter pruning · p. 15

C Practical value of our findings

• With the penalty, VGG-16 and AlexNet maintain performance at higher pruning ratios than without it when pruned using DepGraph.The comparison is shown for Top-5 accuracy on TinyImageNet.Source: Figure 8 · p. 16, C Practical value of our findings · p. 16

D Details of extracting the sparsest AND-OR interactions

• The extraction procedure decomposes masked-sample outputs into AND- and OR-explained components, with learnable γL values determining the decomposition and a LASSO-like loss encouraging sparse interactions.The pseudocode outlines the core procedure for extracting AND-OR interactions.Source: D Details of extracting the sparsest… · p. 17
• The algorithm takes an input sample and DNN, evaluates masked samples, computes AND and OR interaction effects, and returns both interaction sets.It computes confidence scores for masked samples before calculating the interaction effects.Source: D Details of extracting the sparsest… · p. 17
• For transferability evaluation, the authors extract stable interactions shared by the original and pruned DNNs, following the algorithm in [4].The cited method addresses interaction sensitivity to parameters {γL}, which can undermine evaluation stability.Source: D Details of extracting the sparsest… · p. 17
• The extraction method learns a small noise term δT to reduce output noise that can substantially affect extracted interactions, especially at high orders.The δL values are constrained to [−ζ, ζ], with ζ = 0.02 · |v(x) − v(x∅)|.Source: D Details of extracting the sparsest… · p. 17

E.1 Training settings

• ResNet-56 and VGG-19 were trained on CIFAR-10 for 200 epochs using SGD with a 0.1 learning rate, 0.9 momentum, and 5 × 10−4 weight decay.The learning rate was reduced by a factor of 0.1 at epochs 120, 150, and 180, with batch size 128.Source: §E.1 · p. 18

E.2 Pruning settings in Section 2.3

• DepGraph pruning ratios rose from 1.2% to 50.2% for ResNet-56 and from 3.6% to 99.2% for VGG-19, without post-pruning finetuning.Other pruning settings followed the original settings in [2].Source: §E.2 · p. 18

E.3 Pruning settings in Section 2.4

• Module-pruning experiments remove either one residual block or randomly selected hidden dimensions, with 3% to 15% of total hidden dimensionality removed.Models include pretrained ResNet and ViT variants and four LLMs; modules are classified as sensitive when pruning raises test cross-entropy above a model-adjusted threshold τ.Source: §E.3 · p. 18, Table 1 · p. 19

E.4 Details about how to calculate interactions for different DNNs

• For image models, the study samples 10 high-significance patches from intermediate feature maps, while LLM interaction extraction uses 100 manually collected sentences and 10 sampled words per sentence.Image-patch scales are 8 × 8 for CIFAR-10 models and 7 × 7 for ImageNet-1k models; LLM test loss is averaged over WikiText2 tokens.Source: §E.4 · p. 19

E.5 Settings of reference DNNs

• Because training reference DNNs from scratch on test samples is impractical, the authors use independently trained classic models as references and align sampled variable locations across architectures.ResNet-56 and VGG-19 reference each other; other reference-model pairings are specified for the ResNet, ViT, and LLM families.Source: §E.5 · pp. 19–20

E.6 Licenses of Used Assets

• The listed datasets, models, and software assets are used under their respective licenses and terms of use.Examples include Apache License 2.0 for ViT models implemented with timm and the stated access terms for ImageNet-1k.Source: §E.6 · p. 20

F Detailed results of the three-phase dynamics of interactions during progressive parameter pruning

• The appendix adds interaction-distribution results after pruning operations and summarizes interaction changes in a table.The distributions cover ResNet-56 and VGG-19.Source: F Detailed results of the three-phase… · p. 21, §F.1 · p. 21

F.1.1 The distribution of interactions encoded by ResNet-56 during progressive parameter pruning

• For ResNet-56, the distributions show removed and preserved interactions across orders at pruning ratios arranged from low to high.Figure 9 presents the distributions at different pruning ratios.Source: Figure 9 · p. 22, §F.1.1 · p. 22

F.1.2 The distribution of interactions encoded by VGG-19 during progressive parameter pruning

• For VGG-19, the distributions likewise show removed and preserved interactions across orders, arranged by ascending pruning ratio.Figure 10 presents these distributions.Source: Figure 10 · p. 23, §F.1.2 · p. 23

F.2 Quantify the changes in interactions during progressive parameter pruning

• The appendix quantifies pruning-related interaction changes using preserved low-/high-order ratios, an offsetting ratio, and a removed-generalizable-interaction ratio.These measures are denoted ρlow/ρhigh, κ, and ρremoved.Source: §F.2 · p. 24

F.2.1 The ratio of preserved low-order/high-order interactions

• Figure 11 presents the ratio of preserved low-order to high-order interactions during progressive pruning across DNNs.The plotted measure is ρlow/ρhigh.Source: §F.2.1 · p. 24, Figure 11 · p. 24

F.2.2 The offsetting ratio of removed interactions

• Figure 12 presents the offsetting ratio κ of removed interactions during progressive pruning across DNNs.The supplied passage identifies the measure and its model/pruning scope but gives no outcome values.Source: §F.2.2 · p. 24, Figure 12 · p. 24

F.2.3 The ratio of removed generalizable interactions

• Figure 13 presents the ratio ρremoved of removed generalizable interactions during progressive pruning across DNNs.The supplied passage identifies the measure and its model/pruning scope but gives no outcome values.Source: §F.2.3 · p. 24, Figure 13 · p. 25

F.3 Analysis of three-phase dynamics of interactions during parameter pruning

• Table 2 summarizes the three-phase dynamics of interaction changes during progressive parameter pruning.Source: Table 2 · p. 25, §F.3 · p. 25

G.1 The distributions of interactions after the pruning of each module

• For ViT-Small, ViT-Base, and ViT-Large, the figures show removed and preserved interaction-order distributions across module prunings, arranged by increasing test-loss change.Each model's results are presented in two subsets.Source: §G.1 · p. 26, Figure 28 · p. 41, §G.1.8 · p. 41

G.2.1 The ratio of preserved low-order/high-order interactions

• Figure 34 tracks the preserved low-order to high-order interaction ratio after pruning sensitive and insensitive modules across DNNs.The appendix identifies this ratio as one of three perspectives for quantifying interaction changes.Source: Figure 34 · p. 47, §G.2.1 · p. 47, §G.2 · p. 47

G.2.2 The offsetting ratio of removed interactions

• Figure 35 tracks the offsetting ratio of removed interaction effects after pruning sensitive and insensitive modules across DNNs.The appendix identifies this ratio as one of three perspectives for quantifying interaction changes.Source: §G.2.2 · p. 47, §G.2 · p. 47, Figure 35 · p. 48

G.2.3 The ratio of removed generalizable interactions

• Figure 36 tracks the ratio of removed generalizable interactions after pruning sensitive and insensitive modules across DNNs.The appendix identifies this ratio as one of three perspectives for quantifying interaction changes.Source: §G.2 · p. 47, Figure 36 · p. 48, §G.2.3 · p. 48

H Distributions of generalizable interaction

• Generalizable interactions are primarily concentrated at lower orders across twelve DNNs, and low-order interactions are more generalizable than high-order ones.The paper classifies orders 1–3 as low-order and orders 4 to n as high-order because nearly all generalizable interactions fall within the first three orders.Source: H Distributions of generalizable… · p. 49, Figure 37 · p. 49

I Offsetting of removed interaction effects

• Removed interactions exhibit strong offsetting effects.The section presents their distributions for ResNet-56 and VGG-19.Source: I Offsetting of removed interaction… · p. 49, Figure 38 · p. 49

J Properties of the AND interaction

• The AND interaction is presented with efficiency, linearity, dummy, symmetry, anonymity, and recursive axioms.The paper says these properties underscore the reliability of AND interactions for explaining a DNN's inference score.Source: J Properties of the AND interaction · p. 50

K Common conditions for sparse interactions

• One sparsity condition excludes extremely high-order interactions, which the paper describes as complex, over-fitted patterns unlikely to be learned by a well-trained DNN in real scenarios.A second condition requires average classification confidence over masked samples to increase monotonically with the number of unmasked variables.Source: K Common conditions for sparse… · p. 51
• A third condition imposes a polynomial lower bound on confidence for fewer unmasked variables; the paper connects this to classifying masked inputs using visible local information.The paper argues that a well-trained model should not produce substantially reduced confidence for masked samples.Source: K Common conditions for sparse… · p. 51

L Empirical validation of sparsity property

• Across networks and tasks, the extracted interaction set was typically no more than 5,000 for 100 samples, or 50 per sample.The experiment computed AND and OR interactions over all 2^n possible input subsets for each sample.Source: L Empirical validation of sparsity… · p. 52

• The AND proof reorganizes sums over nested subsets to establish universal matching for outputs on masked samples.Source: M Proof of Equation 2 · p. 53
• For OR interactions, the proof swaps summation order and separately handles four cases for L relative to S and N\S.Source: M Proof of Equation 2 · p. 53
• AND-OR interaction terms universally match the model output on masked inputs, with the empty-input output assigned to AND interactions.Source: M Proof of Equation 2 · pp. 53, 55

N Illustration of And-Or logical model

• For the mask prompt, both models predict “settings”; for the transformer prompt, both predict “transformer.”Source: Figure 40 · p. 56, Figure 41 · p. 57, Figure 42 · p. 57
• At pruning ratio 41.01, the original and pruned model both predict “carbon dioxide” and “bonds” on the illustrated prompts.Source: Figure 44 · p. 58, N Illustration of And-Or logical model · p. 58, Figure 45 · p. 59

O Declaration of Large Language Model usage

• The authors say LLMs were used only to polish grammar and clarity, not for ideation, experiment design, data analysis, or substantive content generation.Source: O Declaration of Large Language Model… · p. 61