Source-linked AI summary
DepGraph: Towards Any Structural Pruning
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang
TL;DR
Structural pruning seeks efficient model acceleration, but architecture-specific grouping schemes do not generalize to arbitrary networks with coupled parameters. DepGraph automatically models dependencies and groups coupled parameters, achieving strong acceleration and performance across CNNs, RNNs, GNNs, Transformers, and other tasks.
Problem
Structural pruning lacks a general grouping mechanism for arbitrary architectures because coupled parameters must be pruned together while remaining consistently unimportant.
Method
DepGraph decomposes networks and automatically models layer or component dependencies to form coupled pruning groups.
Results
Across diverse architectures and tasks, DepGraph achieved competitive performance, including 2.57× acceleration with 93.64% accuracy on CIFAR ResNet-56 and 8× to 16× acceleration on non-image models without significant performance drop.
Takeaways & Limitations
DepGraph improves the generalizability of structural pruning across CNNs, RNNs, GNNs, and Vision Transformers through automatic parameter grouping.
Abstract
from arXiv · showhide
Structural pruning enables model acceleration by removing structurally-grouped parameters from neural networks. However, the parameter-grouping patterns vary widely across different models, making architecture-specific pruners, which rely on manually-designed grouping schemes, non-generalizable to new architectures. In this work, we study a highly-challenging yet barely-explored task, any structural pruning, to tackle general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. The most prominent obstacle towards this goal lies in the structural coupling, which not only forces different layers to be pruned simultaneously, but also expects all removed parameters to be consistently unimportant, thereby avoiding structural issues and significant performance degradation after pruning. To address this problem, we propose a general and {fully automatic} method, \emph{Dependency Graph} (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a simple norm-based criterion, the proposed method consistently yields gratifying performances.
1. Introduction
Structural pruning can accelerate networks but is difficult to generalize because coupled parameters require coordinated removal across layers and architectures. DepGraph addresses this by automatically modeling dependencies and grouping parameters, achieving strong results across diverse models and tasks.
- Structural pruning physically removes grouped parameters, reducing network size and computational costs without relying on specific accelerators or software.
- Coupled modules and connections force simultaneous pruning across layers; residual connections, Transformers, RNNs, and GNNs exhibit this dependency.
- Architecture-specific methods require labor-intensive, manually designed dependency analyses that do not directly transfer across network families.
- DepGraph automatically models interdependencies and captures chain effects from pruning one layer to adjacent layers.
- Group-level importance criteria address the fact that single-layer importance becomes unreliable when parameterized layers are structurally entangled.
- 2.57× acceleration with 93.64% accuracy was achieved for ResNet-56 on CIFAR, versus 93.53% for the unpruned model.
- The method achieved more than 2× speed-up on ImageNet-1k ResNet-50 with only 0.32% performance loss, and 8× to 16× acceleration on several non-image models without significant performance drop.
- DepGraph improves structural-pruning generalizability across CNNs, RNNs, GNNs, and Vision Transformers through automatic parameter grouping.
2. Related Work
Related work addresses dependencies among grouped parameters, but complex or nested structures make manual, architecture-specific analysis increasingly labor-intensive. These dependencies require coupled parameters to be pruned simultaneously.
- Structural pruning removes parameter groups, while unstructured pruning zeroes individual weights without changing network structure.
- Dependencies among parameter groups require simultaneous pruning in complex network structures.
- Pruning a filter from one convolutional layer requires removing associated kernels in the following layer.
3. Method
DepGraph provides a fully automatic way to model dependencies between network components, group coupled parameters, and support structural pruning across complex architectures. It decomposes networks into finer components, models inter- and intra-layer dependencies, recursively propagates relationships, and uses group-level sparsity for importance estimation.
- Dependency Graph: DepGraph addresses the difficulty of manually analyzing nested and composed parameter dependencies in arbitrary neural networks.The method is designed as a general and fully automatic mechanism for dependency modeling.
- Grouping: A grouping matrix G records dependencies between layers, with g(i) identifying all layers coupled to layer i.Self-dependency is enabled through the diagonal of G.
- Dependency Graph: Recursive propagation compresses redundant dependency relations while retaining the transitive coupling needed to form pruning groups.Starting from one parameter, the procedure expands through newly discovered dependencies until the group is complete.
- Network Decomposition: Network decomposition represents each component through its inputs and outputs, allowing dependencies to be modeled even when one layer has different input and output pruning schemes.Components may be parameterized layers or non-parameterized operations such as residual addition.
- Dependency Modeling: Inter-layer dependencies arise from network connectivity, while intra-layer dependencies arise when a component’s inputs and outputs share a pruning scheme.Convolution inputs and outputs can follow distinct schemes, whereas batch normalization can require simultaneous input-output pruning.
- Group-level Pruning: Group-level sparse training forces coupled parameters toward consistent zeros, enabling a simple norm-based criterion to identify removable groups.The regularization strength γ_k is controlled using the importance of each prunable dimension and an exponential strategy.
4. Experiments
Experiments evaluate DepGraph across image, text, 3D point-cloud, and graph tasks, testing grouping, sparsity, generalizability, and transfer across architectures. Results show that full group-aware sparse training, learned sparsity, and dependency modeling support effective pruning across diverse networks.
- Experimental setup: DepGraph is evaluated on CIFAR, ImageNet, AGNews, ModelNet, and PPI using CNNs, Transformers, LSTMs, DGCNNs, and GATs.The evaluated architectures include ResNe(X)t, VGG, DenseNet, MobileNet, GoogleNet, Vision Transformers, LSTM, DGCNNs, and Graph Attention Networks.
- Grouping Strategy: Full grouping accounts for all parameterized layers during sparse training, while competing strategies use only partial layers.The ablation compares no grouping, convolution-only grouping, and full grouping across grouping and sparsity configurations.
- Grouping Strategy: Grouping partial parameters improves final performance, while full grouping can further improve pruning accuracy by using information from all grouped layers.The proposed strategy consistently promotes sparsity across convolutions, batch normalizations, and fully-connected layers.
- Learned Sparsity: Learned sparsity usually outperforms uniform sparsity because different layers are not equally prunable, although over-pruning can degrade accuracy.The paper notes that learned sparsity may reduce some layers excessively in certain cases.
- Generalizability of DepGraph: DepGraph handles DenseNet and GoogleNet’s dense and parallel connections and derives coupled parameter groups for complex networks automatically.Visualizations cover DenseNet-121, ResNet-18, and Vision Transformer, with large groups observed within DenseNet dense blocks.
- Towards Any Structural Pruning: A simple norm-based criterion combined with dependency modeling achieves performance comparable to methods using stronger criteria and training techniques.The claim is made for ImageNet pruning across ResNet, DenseNet, MobileNet, ResNeXt, and Vision Transformers.
- Towards Any Structural Pruning: Experiments on text, graphs, and 3D point clouds indicate that DepGraph generalizes beyond image architectures.The non-image evaluations include LSTM text classification, Dynamic Graph CNNs, and Graph Attention Networks.
5. Conclusion
Dependency Graph enables any structural pruning across a wide variety of neural network architectures.
- Dependency Graph is introduced as a general algorithm for structural pruning across CNNs, RNNs, GNNs, and Transformers.