PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Network Pruning with Residual-Connections and Limited-Data

Jian-Hao Luo, Jianxin Wu

arXiv:1911.08114v3cs.CV

TL;DR

Pruning residual connections and pruning with limited data remain open challenges, while direct pruning on small datasets is usually weaker than fine-tuning a small model prepared with large-scale data. CURL prunes channels inside and outside residual connections and combines data augmentation, knowledge distillation, and label refinement; it reports improved benchmark performance, including comparable or better results when pruning on small datasets.

  • Problem

    Pruning residual connections and pruning models with limited data are open issues, and existing pruning methods often omit the small-dataset application setting.

    Source: §1 · p. 1

  • Method

    CURL prunes channels inside and outside residual connections using a KL-divergence criterion and uses image transformations, knowledge distillation, and label refinement for limited-data fine-tuning.

    Source: §1 · p. 2, §3.1.2 · p. 4, §3.2 · p. 4

  • Results

    CURL outperforms previous state-of-the-art methods on ImageNet and achieves comparable or better results than fine-tuning a small model prepared on large-scale data when pruning on small datasets.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    CURL can directly prune models on small datasets while matching or exceeding fine-tuning a small model pruned or trained on large-scale data.

    Source: §5 · p. 8

    The paper uses a simple method to process noisy soft targets and notes that more advanced correction algorithms may further improve accuracy.

    Source: §4.3.3 · p. 8

Abstract

from arXiv · show

Filter level pruning is an effective method to accelerate the inference speed of deep CNN models. Although numerous pruning algorithms have been proposed, there are still two open issues. The first problem is how to prune residual connections. We propose to prune both channels inside and outside the residual connections via a KL-divergence based criterion. The second issue is pruning with limited data. We observe an interesting phenomenon: directly pruning on a small dataset is usually worse than fine-tuning a small model which is pruned or trained from scratch on the large dataset. Knowledge distillation is an effective approach to compensate for the weakness of limited data. However, the logits of a teacher model may be noisy. In order to avoid the influence of label noise, we propose a label refinement approach to solve this problem. Experiments have demonstrated the effectiveness of our method (CURL, Compression Using Residual-connections and Limited-data). CURL significantly outperforms previous state-of-the-art methods on ImageNet. More importantly, when pruning on small datasets, CURL achieves comparable or much better performance than fine-tuning a pretrained small model.

1. Introduction

The introduction identifies two challenges: pruning residual connections and pruning with limited data. CURL addresses both by pruning channels inside and outside residual connections and using augmentation, distillation, and label refinement for limited-data fine-tuning.

1. Introduction

  • The wallet structure is more accurate, faster at the same FLOPs, and saves more storage than the hourglass structure.The stated accuracy advantage is attributed to a larger pruning space; storage savings come from pruning more weights.Source: §1 · p. 1
  • Direct pruning on a small dataset usually underperforms fine-tuning a small model pruned or trained on a large dataset, although real applications may lack access to large datasets.The introduction gives CUB200 accuracy of 66.90% for direct pruning versus 69.43% for fine-tuning a pruned ImageNet model.Source: §1 · pp. 1–2
  • CURL outperforms previous state-of-the-art methods on ImageNet and its techniques improve accuracy when pruning with limited data.The approach combines residual-connection pruning with knowledge distillation, mixup, image transformations, and label refinement.Source: §1 · p. 2
  • CURL prunes channels inside residual branches and output activations, producing a wallet-shaped structure rather than the hourglass shape of inside-only pruning.The output activations include both identity and residual branches.Source: §1 · pp. 1–2
  • Combining data augmentation and knowledge distillation improves limited-data fine-tuning, while label refinement is proposed to reduce the effect of noisy teacher logits.Mixup is named as an augmentation technique.Source: §1 · p. 2

2. Related Work

Prior pruning work largely targeted toy or large-scale datasets and generally left residual blocks hourglass-shaped. The paper positions limited-data pruning and output-channel pruning as gaps in that work.

2. Related Work

  • Non-structured pruning removes individual weights but can impair actual inference speed through irregular connections and memory access, motivating structured pruning.Structured filter pruning discards whole filters judged unimportant.Source: §2 · p. 2
  • Filter-importance approaches use magnitude, LASSO with least-squares reconstruction, next-layer statistics, or within-layer geometric medians.Other methods select channels through batch-normalization scaling, channel-selection layers, reinforcement learning, or slimmable networks.Source: §2 · p. 2
  • Except for slimmable networks, the cited filter-pruning methods produce hourglass-shaped models, leaving residual-block output dimensions unchanged.CURL’s Figure 2 illustrates pruning both internal channels and block outputs.Source: §2 · p. 2, Figure 2 · p. 3
  • Prior methods achieved results on toy or large-scale datasets but ignored pruning with limited data; the authors describe their work as the first attempt to address that setting.The cited benchmark scales include MNIST, CIFAR, and ImageNet.Source: §2 · p. 3

3. Our Method

CURL combines residual-connection pruning with pruning under limited data. It uses a global KL-divergence criterion and a data-expansion and distillation pipeline with label refinement.

3.1.1 Recap of Residual-Connections

  • Stage-wide pruning removes channels inside and outside residual paths, expanding pruning space; the paper says this is faster at equal FLOPs and saves activation-map memory.Source: §3.1.1 · p. 3

3.1.2 The Proposed Pruning Method

  • CURL scores filters by KL divergence between original and channel-removed network output probabilities, enabling global, adaptive pruning across layers.Source: §3.1.2 · pp. 3–4

3.2. Prune with Limited Data

  • For limited data, CURL expands images and fine-tunes the pruned small model with knowledge distillation; potentially noisy teacher logits motivate label refinement.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 Data Expansion

  • Data expansion uses random rotation, cutout, and patch shuffling to emphasize local discriminative image regions rather than global context.Source: §3.2.1 · p. 4

3.2.2 Label Refinement

  • Training first uses knowledge distillation with mixup on original data, then expanded-data distillation with reversed KL and soft-target updates to refine noisy teacher logits.Source: §3.2.2 · p. 5

4. Experiments

The experiments evaluate CURL on ImageNet, four small-scale datasets, and ablations of its fine-tuning strategy, pruning criterion, and label refinement. They report stronger ImageNet pruning results, competitive small-dataset performance, and gains from knowledge distillation and label refinement.

4.1. Pruning ResNet50 on ImageNet

  • CURL is reported to outperform prior state-of-the-art on ImageNet, with higher accuracy despite fewer MACs and parameters.The comparison concerns ResNet50 pruning; Table 1 presents the ImageNet pruning results.Source: Table 1 · p. 6, §4.1 · p. 6
  • At matched 1.39G MACs and 7.83M parameters, the wallet model processes 256 images in 0.19s versus 0.21s for hourglass, with nearly half the model size.The reported inference timing is on an NVIDIA Tesla M40 GPU.Source: §4.1 · p. 6

4.2. Pruning on Small-Scale Datasets

  • On four small datasets, CURL is comparable to or better than ImageNet-pretrained small models; MobileNetV2-1.0 CURL beats MobileNetV2-0.5 throughout.For ResNet, direct pruning is less advantageous against the strong ResNet50-CURL baseline, but performs better on CUB200-2011 and Oxford Car.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3.1 Impact of Fine-Tuning Strategy

  • 78.72% accuracy with mixup, knowledge distillation, and label refinement nearly matches the unpruned large model's 78.77%; training the pruned network from scratch reaches only 64.39%.The scratch model was trained for 1,000 epochs with mixup; fine-tuning with knowledge distillation alone reached 77.87%, and mixup plus distillation reached 77.99%.Source: §4.3.1 · p. 7

4.3.2 Impact of Pruning Criterion

  • 66.33% accuracy with the KL-divergence criterion exceeds the reported 21.83% for loss change and 8.25% for extended weight sum.The extended weight-sum score was only slightly better than the random-pruning result of 6.80%.Source: §4.3.2 · p. 8

4.3.3 Impact of Label Refinement

  • Label refinement improves on mixup-equipped standard knowledge distillation across most datasets.The method enlarges the training data sixfold and updates soft targets during distillation; on expanded CUB200, updating them yields 78.72% versus 78.56% with η = 0.Source: §4.3.3 · p. 8

5. Conclusion

CURL prunes channels inside and outside residual connections using a KL-divergence criterion and uses label refinement to address label noise. On small datasets, it can achieve comparable or better results than fine-tuning a small model pruned or trained on large-scale data.

5. Conclusion

  • On small datasets, CURL achieves comparable or better results than fine-tuning a small model pruned or trained on large-scale data.The comparison is reported for direct pruning with CURL on the target small dataset.Source: §5 · p. 8
  • CURL prunes channels both inside and outside residual connections using a KL-divergence-based criterion.Source: §5 · p. 8
  • CURL uses label refinement to avoid the influence of label noise.Source: §5 · p. 8

Related papers

Loading 1911.08114v3…