Source-linked AI summary
Breaking the Compression Barrier: Cross-Architecture Compression Boundary Learning via Reverse Regrowth
Zhaocen Liu, Satvik Praveen, Yi Sheng
TL;DR
Extreme sparsity can trigger abrupt accuracy collapse, making feasible compression boundaries difficult to identify. BRIDGE over-compresses models to expose that boundary, then selectively regrows critical structures, restoring usable performance across CNNs and Transformers with gains up to 4.77%.
Problem
Abrupt performance cliffs under extreme sparsity make feasible compression boundaries difficult to identify with conventional forward pruning.
Method
BRIDGE intentionally over-compresses models to expose the collapse boundary, then hierarchically regrows only critical parameters or channels.
Results
Gains of up to 4.77% were achieved while restoring near-collapsed models to usable performance across CNN and Transformer architectures.
Takeaways & Limitations
Reverse regrowth provides an architecture-agnostic approach for identifying compression boundaries and recovering usable performance without additional inference overhead.
Abstract
from arXiv · showhide
Model compression is critical for deploying networks on resource-constrained edge devices. While pruning-based methods can significantly reduce model size, they often suffer from abrupt performance collapse beyond a sparsity thresh-old, making it difficult to identify the feasible compression limit of the model. To address this challenge, we propose a boundary-Learning reverse regrowth framework, BRIDGE, that reformulates compression as a constructive boundary-search problem. Unlike forward pruning, our method first drives the model to an extremely sparse state to expose the collapse region, and then selectively regenerates the critical structure to restore performance. The proposed framework employs a hierarchical regeneration strategy, including coarse-grained layer selection and fine-grained regeneration parameter selection, to accurately identify which parameters require recovery. Experiments show that our method can recover models from the brink of collapse on both CNNs and Transformer architectures, demonstrating its architecture in-dependence. BRIDGE achieves a performance improvement of up to 1.49% in unstructured pruning and up to 4.77% in structured pruning. These results demonstrate that reverse regeneration can effectively extend the compression limit while maintaining stable performance. The source code is available at https://github.com/EnumaCaliber/BRIDGE.
1 Introduction
The introduction frames maximum feasible compression as an unresolved boundary-estimation problem because forward pruning can abruptly collapse in performance. BRIDGE addresses this by crossing the collapse boundary and selectively regrowing critical parameters through hierarchical, RL-guided recovery.
- Motivation: Maximum feasible compression remains unanswered because pruning methods optimize accuracy under predefined sparsity budgets rather than identifying the collapse limit.The problem applies to both structured and unstructured pruning settings.
- Problem: Forward pruning progressively compresses dense models in a large-to-small paradigm, but performance cliffs make this approach ill-suited for locating the compression boundary.The limitation is associated with the monotonic forward-sparsification process rather than a particular pruning criterion.
- Observations: Performance cliffs occur across one-shot and iterative pruning and are highly localized, indicating that compression boundaries are intrinsic to models and concentrated in specific structures.The observation spans representative convolutional and Transformer architectures.
- BRIDGE: BRIDGE reformulates compression as boundary learning by intentionally traversing the compression boundary, then selectively restoring the minimum critical subset of pruned parameters.This reverse-regrowth strategy explicitly exposes the collapse region before recovery.
- Method: BRIDGE uses an RL-based controller with hierarchical regeneration: SSIM filtering selects degraded layers, while gradient saliency ranks critical pruned weights within them.The two-level strategy allocates a limited parameter budget while restricting recovery to the most structurally degraded regions.
2 Related Works And Motivation
The section shows that pruning across architectures and strategies exhibits a distinct performance cliff, while collapse is concentrated in a small subset of layers. It motivates BRIDGE’s reverse-regrowth approach, which over-compresses models before restoring critical structures to locate the feasible boundary.
- Observation 1: Performance Cliffs: Performance cliffs consistently emerge across pruning strategies and architectures, including VGG16 and ViT.The evaluation covers one-shot and iterative pruning, with structured and unstructured settings for VGG16 and structured pruning for ViT.
- Motivation 1: Revisiting Forward Sparsification: Forward sparsification progressively removes parameters from dense models while preserving accuracy, but its effectiveness is limited under extreme sparsity.Moderate sparsity benefits from abundant redundancy and generally sufficient local pruning decisions.
- Observation 2: Performance Collapse Is Highly Localized: Performance collapse is highly localized, with severe representational disruption occurring in only a few layers while most of the network remains stable.VGG16 shows dramatic post-collapse sign flips and low SSIM values; analogous behavior is observed in ViT.
- Motivation 2: Bridging Pruning and Representation Analysis: Recovering a model beyond the compression boundary requires determining both what to modify and where recovery should occur.Existing pruning and representation-analysis methods address parameter importance and recovery location separately.
- Motivation 3: From Sparse Maintenance to Boundary Recovery: BRIDGE differs from sparse-network maintenance by over-compressing models to expose the boundary, then hierarchically regrowing critical structures for boundary recovery.Its objective is recovery beyond the safe operating regime rather than maintaining functionality during sparse training.
3 Framework
BRIDGE frames compression boundary learning as a three-stage process: identify collapse, iteratively regrow critical capacity, and select a model meeting the target performance threshold. Its hierarchical regrowth combines layer-level selection with fine-grained restoration guided by higher-order importance.
- Framework overview: BRIDGE’s pipeline consists of three stages: collapse-region identification, iterative reverse regrowth, and deployment-ready model selection.The framework details each stage in dedicated subsections.
- Stage I: Collapse Region Identification: BRIDGE progressively prunes models to increasing sparsity levels using iterative or one-shot pruning to establish a baseline and identify collapse.Its regrowth procedure is order-independent, requiring an importance measure of higher order than the pruning criterion.
- Stage I: Collapse Region Identification: The collapse onset is the smallest sparsity s∗ where the discrete second-order accuracy difference falls below −ε, with ε set to 0.5 experimentally.This criterion detects the transition from approximately linear degradation to sharp nonlinear collapse.
- Stage II: Reverse Regrowth: Starting near collapse, an RL controller iteratively observes the model state and chooses hierarchical actions to restore capacity, followed by fine-tuning and evaluation.The loop is formulated as a Markov Decision Process, with evaluation producing rewards for policy updates.
- Stage II: Reverse Regrowth: Hierarchical reverse regrowth first selects a target layer coarsely, then restores the highest-ranked parameters or channels using a higher-order importance measure.SSIM identifies degradation-sensitive layers where applicable, while cosine similarity covers layers with small or absent spatial dimensions.
- Stage III: Deployment: After Stage II converges, BRIDGE selects a regrown model satisfying the target performance threshold as the final deployment-ready output.Deployment evaluation uses structured-pruned models because structured sparsity directly provides hardware benefits.
4 Experiments
BRIDGE is evaluated across CNN and Transformer architectures, datasets, and pruning settings using a common fine-tuning protocol. It consistently recovers performance near collapse boundaries while reducing search costs and improving deployment trade-offs.
- Experimental setup: Experiments cover ResNet20, EfficientNetB0, ShuffleNetV2, VGG16, and ViT across CIFAR-10 and Tiny-ImageNet.Unless otherwise specified, fine-tuning uses AdamW with learning rate 3×10−4, weight decay 10−2, early stopping, and full-test-set accuracy evaluation.
- Performance near collapse: +4.77% improvement is achieved by BRIDGE on ResNet20 under structured pruning near the collapse boundary.On VGG16, the BRIDGE-recovered curve trends upward at high sparsity while the baseline declines sharply.
- Performance near collapse: BRIDGE lifts structured-pruned VGG16 from below 80% accuracy back above 80%, close to the low-sparsity baseline.The improvements concentrate where performance shifts from gradual degradation to abrupt failure and baselines fall below critical usability thresholds.
- Edge deployment: 43.2× speedup is reached on VGG16, while BRIDGE simultaneously achieves higher sparsity, higher accuracy, and lower latency than forward pruning across three architectures.The deployment results are reported on Raspberry Pi 5 under structured pruning.
- Regrowth analysis: +4.48% improvement is obtained on ViT with second-order regrowth applied to a first-order pruned model at 69.48% sparsity.Both first- and second-order regrowth outperform corresponding pruning baselines on VGG16 and ResNet20, supporting the order-based criterion across architectures.
5 Conclusion
BRIDGE reframes extreme compression as boundary-search: it over-compresses models to expose collapse, then selectively regrows critical structures. Across CNNs and Transformers, it restores near-collapsed models without additional inference overhead and achieves gains of up to 4.77%.
- 5 Conclusion: BRIDGE over-compresses models to expose the compression boundary, then selectively recovers critical structures through hierarchical regrowth.The framework reformulates compression as a constructive boundary-search problem rather than purely destructive pruning.
- 5 Conclusion: Up to 4.77% gains are achieved while restoring near-collapsed models to usable performance without additional inference overhead.
- 5 Conclusion: BRIDGE delivers consistent improvements across both CNN and Transformer architectures, demonstrating architecture-agnostic compression recovery.