BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks

1 Introduction

• Unstructured pruning removes individual weights and can reach higher sparsity than structured pruning, which removes whole components.Gradual pruning uses a schedule with retraining between steps so the network can recover as weights are removed [Zhu and Gupta, 2017, Gale et al., 2019].Source: §1 · p. 1
• At extreme sparsity, accuracy degrades sharply as learned representations become degraded and unstable, making subsequent pruning decisions less reliable.The passage gives 99.9% sparsity as an example.Source: §1 · p. 1
• BaCP adds supervised and self-supervised contrastive terms that align sparse-model representations with pretrained, fine-tuned, and intermediate-snapshot references.Its design builds on CAP’s PrC/FiC/SnC decomposition, previously applied to BERT-family encoders up to 97% sparsity [Xu et al., 2022].Source: §1 · p. 1
• BaCP is evaluated on five convolutional backbones across CIFAR-10 and CIFAR-100, using Magnitude, SNIP-it, and WANDA criteria up to 99.95% sparsity.Each run is matched to an iterative-pruning baseline; a cross-entropy control holds the learning-rate and augmentation changes fixed to separate their contribution from the objective’s.Source: §1 · p. 2

2 Methodology

• The masked backbone produces features for a linear classification head and a normalized projection-head embedding, while the projection head is discarded before evaluation.The sparse model’s projection head is trainable and unpruned; anchor heads are frozen at initialization and are not restored from anchor checkpoints.Source: §2 · p. 2

2.1 Anchored Contrastive Alignment

• BaCP stacks sparse-model and frozen-anchor embeddings into a 2B × p matrix and computes contrastive losses over the full 2B × 2B similarity matrix.The instance-level positives use the other view of the same input, while label-level positives include samples with the same class [Chen et al., 2020, Khosla et al., 2020]; frozen anchors receive no gradient.Source: §2.1 · p. 2

2.2 Multi-Source Anchors

• BaCP uses pretrained, fine-tuned, and earlier sparse snapshots as frozen anchors, respectively supplying general features, class-discriminative structure, and consistency with recent model history.The snapshot loss uses a mean to keep its scale and λSnC independent of K; the experiments retain K = 2.Source: §2.2 · pp. 2–3

2.3 Overall Objective and Pruning Pipeline

• The sparse model minimizes cross-entropy plus three anchored losses, with all λ weights set to 0.25 and ablation weights renormalized to sum to one.Magnitude, WANDA, and SNIP-it scores are globally thresholded; sparsity is measured over pruned convolutional and linear layers, including the task head, while biases, normalization parameters, and projection heads are exempt.Source: §2.3 · p. 3
• A cubic schedule ramps sparsity over the first 80% of steps and holds the target for the final 20% recovery phase, with masks recomputed every 100 optimizer steps.Masks are reapplied after optimization, so pruned weights are not regrown.Source: §2.3 · p. 3
• The iterative-pruning baseline shares the initialization, checkpoint, criterion, schedule, prunable scope, data splits, and epoch budgets, but differs in pruning-phase learning rate and augmentation.A cross-entropy control uses BaCP’s learning rate and augmentation settings while zeroing the contrastive weights; SNIP-it may also select different masks because its score depends on the training loss.Source: §2.3 · p. 3

3 Results

• BaCP leads in 53 comparisons across the 90-cell grid and a ResNet-34 CIFAR-100 sweep, with gains concentrated where pruning damages the baseline.Table 1 reports held-out CIFAR-10 test accuracy as BaCP = I.P. + ∆; entries are means over three seeds.Source: §3 · pp. 3–4, Table 1 · p. 4
• Below 70% baseline accuracy, BaCP improves 91% of live cells by a median +4.76 points; above 85%, the median is +0.01 and the win rate is 50%.The comparison excludes seven collapsed chance-level cells; classifying cells on seed 1 and measuring margins on seeds 2 and 3 leaves the effect intact.Source: §3 · p. 4
• In the damaged regime, augmentation and learning-rate changes contribute +6.62 points against +0.77 for contrastive terms; in the intact regime, augmentation costs −2.08 and the objective returns +1.15.The control holds BaCP’s augmentation learning rate and schedule while zeroing the three contrastive weights.Source: §3 · p. 4
• A control attributes the MobileNetV2 regression of −1.77 to −5.63 points at 95% sparsity to the augmentation change; projection-head capacity does not explain it.The snapshot schedule leaves the result unchanged.Source: §3 · p. 4

4 Conclusion

• Below 70% baseline accuracy, BaCP improves 91% of cells by a median 4.76 points, while above 85% it is indistinguishable from baseline.The conclusion attributes most of the gain to augmentation and learning-rate changes rather than the objective.Source: §4 · p. 5

A Appendix contents

• The appendices provide full result and ablation tables, negative and refuted results, run protocols, reproducibility details, compute and asset information, and broader-impact discussion.They also include the full algorithm and per-cell surviving-parameter counts.Source: A Appendix contents · p. 7

B Algorithm

• BaCP combines pretrained-anchor, fine-tuned-anchor, and snapshot contrastive terms with cross-entropy, while I.P. and CE-only use cross-entropy weights.The three arms share the procedure; they differ in objective settings and, for I.P., view count.Source: B Algorithm · p. 8
• The pretrained anchor starts from ImageNet weights, the fine-tuned anchor is the dense task checkpoint, and snapshots are saved during pruning.The snapshots are frozen copies of the sparse model from earlier pruning stages.Source: B Algorithm · p. 8
• Stage 3 applies contrastive and task losses during gradual pruning, then re-imposes the mask after each optimizer step.Mask updates use a global step clock; the sparsity ramp ends at 80% of total steps.Source: B Algorithm · pp. 8–9
• Stage 4 freezes the mask for fine-tuning, with the projection head removed from the graph and logits used for cross-entropy.The head is not evaluated or pruned in this stage.Source: B Algorithm · p. 8
• With K = 2, BaCP uses four forwards per step versus I.P.’s one, although both receive 50 pruning and 25 fine-tuning epochs.The comparison matches epoch budgets, not wall-clock time.Source: B Algorithm · p. 9

C Full result tables

• The full tables report absolute top-1 accuracy as mean ± sample standard deviation over three seeds and list surviving prunable weights.The surviving-weight count distinguishes architectures that share a sparsity ratio.Source: C Full result tables · p. 9
• On ResNet-34 CIFAR-100, margins are near zero when the baseline is intact and large once it is not.The caption identifies this as a reproduction of the Table 1 pattern on a second dataset and label space.Source: Table 2 · p. 9
• Table 3 covers every grid cell for both I.P. and BaCP under a matched 50+25-epoch budget, with three seeds per arm.The table description specifies coverage and protocol, but not cell-level outcomes.Source: Table 3 · p. 9

D.1 Objective ablation, every cell

• The CE-only control zeroes the contrastive terms while retaining BaCP settings, decomposing the margin into augmentation/learning-rate and objective contributions.The table description covers all cells and three seeds per arm.Source: §D.1 · p. 11, Table 4 · p. 11

D.2 Projection head

• The projection-head comparisons include narrower heads, a frozen head, and no head; the head is not pruned and is discarded at evaluation.The supplied table text identifies model, sparsity, accuracy, seed, and control columns, but gives no outcomes.Source: §D.2 · p. 12

D.3 Snapshot schedule

• The snapshot comparison contrasts snapfirst2, which fills the cap in opening epochs, with snapevenK, which spreads snapshots across pruning.The table header indicates comparison against a control but supplies no results.Source: §D.3 · p. 12

D.4 VGG learning rate

• The table lists I.P. at 0.01 and BaCP at 0.05 and 0.1, including a 0.1-versus-0.05 comparison; entries use three seeds, and the protocol measures rather than asserts its architecture-specific choice.Source: §D.4 · p. 12

E Negative and refuted results

• Above an 85% baseline, BaCP’s median margin is approximately zero and its win rate is indistinguishable from chance across 46 cells.The paper presents this intact-regime result as a scope condition; it covers half the grid.Source: E Negative and refuted results · p. 12
• Nine projection-head comparisons across two sparsities did not improve over the default, despite the head reaching 744% of the surviving backbone on MobileNetV2 at 0.99.The passage says the capacity ratio is real but not causal for the regression.Source: E Negative and refuted results · p. 13
• Three alternative snapshot schedules at two sparsities changed results by at most a few tenths of a point, with no consistent direction.Source: E Negative and refuted results · p. 13
• Raising VGG-11’s pruning-phase learning rate from 0.05 to 0.1 reduced accuracy at both tested sparsities and increased seed spread by an order of magnitude.Source: E Negative and refuted results · p. 13
• Cells where both arms are at or near chance are excluded from every aggregate because their margin is not interpretable.Source: E Negative and refuted results · p. 13

F Protocol as run

• The protocol uses ImageNet-pretrained initialization, a shared dense checkpoint, cubic pruning over 80% of steps, and 50 pruning plus 25 mask-frozen fine-tuning epochs.The listed protocol also specifies batch size 512, three criteria, and a matched prunable scope.Source: F Protocol as run · p. 13
• BaCP uses two SimCLR views, objective weights of (.25, .25, .25, .25), temperature 0.15, and two greedy snapshots.The protocol lists pruning-phase SGD rates of 0.1 for BaCP and 0.05 on VGGs, versus 0.01 for I.P.Source: F Protocol as run · p. 13

• Every cell used seeds 1, 2, and 3 on both arms, except head and snapshot ablations, which were labeled and used only to refute mechanisms.Source: G Reproducibility · p. 13
• Contrastive-phase selection used training loss, both fine-tunes used validation accuracy, and the test set was not used for selection.Source: G Reproducibility · p. 13
• Runs were admitted only when recorded settings matched the protocol, including epochs, sparsity, criterion, seed, and verified dense-checkpoint loading.An automated recorded-field check excluded mismatches, including runs from an earlier generation that failed to load the dense checkpoint.Source: G Reproducibility · pp. 13–14

Compute

• At K = 2, BaCP uses four forward passes per step versus one for I.P., so matched epochs and schedules do not mean matched GPU time.Runs were serialized on one NVIDIA A100; the project also used unreported compute for discarded runs and one-seed protocol probes.Source: H Compute, assets and broader impact · p. 14

Existing assets

• The experiments use CIFAR-10 and CIFAR-100 and ImageNet-pretrained torchvision ResNet-34/50, VGG-11/19, and MobileNetV2 checkpoints; no new dataset or model is released.The datasets are used as distributed and unmodified; the checkpoints and PyTorch are used under BSD-3-Clause licenses.Source: H Compute, assets and broader impact · p. 14

Broader impact

• The paper reports aggregate top-1 accuracy only, so it does not establish whether BaCP protects under-represented groups from faster pruning-related accuracy loss.The authors note that cheaper inference can lower deployment costs, while pruning may degrade under-represented classes faster than aggregate accuracy suggests.Source: H Compute, assets and broader impact · p. 14

Claims and scope

• The abstract and introduction accurately scope BaCP’s claims: its advantage is large when pruning damages the baseline and close to zero otherwise, supported by 53 of 90 cells.A cross-entropy control attributes only about a tenth of the damaged-regime margin to the objective itself.Source: NeurIPS Paper Checklist · p. 15

Limitations

• The paper reports no benefit above an 85% baseline, no helpful regime for VGG-11, regression for MobileNetV2, and aggregate top-1 reporting that misses per-class degradation.A cross-entropy control suggests roughly nine tenths of the damaged-regime margin comes from augmentation and learning-rate changes rather than the objective; Appendix E collects negative and refuted results.Source: NeurIPS Paper Checklist · p. 16

Reproducibility and access

• The paper describes an anonymized code repository, public datasets and checkpoints, and full protocol and hyperparameter details, but does not claim bit-for-bit reproduction.Reproduction information includes the objective, criteria, pruning scope and schedule, matched-baseline protocol, pseudocode, per-arm hyperparameters, seeds, selection signals, and run-admission rules.Source: NeurIPS Paper Checklist · pp. 17–18

Experiment statistical significance

• Each main-grid cell uses seeds 1, 2, and 3, with means and sample standard deviations; head and snapshot ablations use one seed, and no significance test is claimed.The median per-cell seed spread is 0.29 points and exceeds 0.3 in 65% of cells.Source: NeurIPS Paper Checklist · p. 18

Experiment compute resources

• Appendix H reports a single NVIDIA A100 on a 24-vCPU cloud node, 540 serialized runs, higher per-step cost for BaCP, and additional compute from discarded runs.A BaCP step forwards four networks against the baseline’s one.Source: NeurIPS Paper Checklist · p. 19

Ethics, assets, and human subjects

• The paper reports using public benchmarks and checkpoints, no human subjects or personal data, no released new assets, and credits and licenses the existing assets.It states that no IRB review was required because the work involved no human-subject research.Source: NeurIPS Paper Checklist · pp. 19, 21, 22

Societal impact

• Pruning may widen device access through cheaper storage and inference, but cheaper deployment and faster degradation of under-represented classes are noted as risks not captured by aggregate top-1.Appendix H discusses both potential positive and negative societal impacts.Source: NeurIPS Paper Checklist · pp. 16, 20

NeurIPS Paper Checklist

• The paper marks LLM use N/A because no LLM is part of BaCP’s method, experiments, or analysis.BaCP is a training objective for convolutional image classifiers, and the reported results come from those training runs.Source: NeurIPS Paper Checklist · p. 22