Source-linked AI summary
Channel-wise Knowledge Distillation for Dense Prediction
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, Chunhua Shen
TL;DR
Dense prediction distillation often relies on spatial alignment, but strict alignment can be unsuitable for pixel-level tasks. This paper instead normalizes corresponding channels into probability maps and minimizes asymmetric KL divergence, reporting consistent gains across benchmarks and tasks.
Problem
Strict point-wise or feature-map alignment can impose overly restrictive constraints for dense prediction, where direct transfer of classification KD may yield unsatisfactory segmentation results.
Method
The method converts each channel’s activations into a probability distribution and softly aligns corresponding teacher and student channels using asymmetric KL divergence.
Results
The method consistently outperforms state-of-the-art distillation methods across four public benchmark datasets, network backbones, semantic segmentation, and object detection.
Takeaways & Limitations
Channel-wise distillation is presented as a simple, effective baseline that can complement spatial distillation and support compact dense-prediction networks.
Takeaways & Limitations
The claimed benefit of KL asymmetry for dense-prediction distillation remains a hypothesis stated by the authors.
Abstract
from arXiv · showhide
Knowledge distillation (KD) has been proven to be a simple and effective tool for training compact models. Almost all KD variants for dense prediction tasks align the student and teacher networks' feature maps in the spatial domain, typically by minimizing point-wise and/or pair-wise discrepancy. Observing that in semantic segmentation, some layers' feature activations of each channel tend to encode saliency of scene categories (analogue to class activation mapping), we propose to align features channel-wise between the student and teacher networks. To this end, we first transform the feature map of each channel into a probabilty map using softmax normalization, and then minimize the Kullback-Leibler (KL) divergence of the corresponding channels of the two networks. By doing so, our method focuses on mimicking the soft distributions of channels between networks. In particular, the KL divergence enables learning to pay more attention to the most salient regions of the channel-wise maps, presumably corresponding to the most useful signals for semantic segmentation. Experiments demonstrate that our channel-wise distillation outperforms almost all existing spatial distillation methods for semantic segmentation considerably, and requires less computational cost during training. We consistently achieve superior performance on three benchmarks with various network structures. Code is available at: https://git.io/Distiller
1. Introduction
Dense prediction distillation is challenging because strict spatial alignment can impose overly restrictive constraints. The paper proposes channel-wise distillation that normalizes each channel into a probability map and aligns teacher–student distributions with asymmetric KL divergence.
- Dense prediction models require strong pixel-level representations but typically incur high computational costs, motivating compact networks trained with knowledge distillation.
- Directly transferring classification distillation to semantic segmentation can be unsatisfactory because strict point-wise or feature-map alignment may produce sub-optimal solutions.
- Existing spatial methods normalize activations at locations and aggregate spatial relationships, but every location contributes equally and may transfer redundant teacher information.
- Channel-wise distillation converts each channel activation map into a probability distribution and minimizes asymmetric KL divergence between corresponding teacher and student channels.
- Channel activations tend to encode scene-category saliency, so the student is guided to mimic highly activated regions, such as foreground objects in detection.
- The method is reported to improve baseline spatial distillation by a large margin and significantly outperform state-of-the-art KD methods across dense prediction tasks.
- The authors report consistent improvements on four benchmark datasets with varied network structures, supporting the method’s generality.
2. Related Work
Prior knowledge-distillation research has focused mainly on classification, while dense prediction requires methods beyond naive pixel-wise transfer. Related segmentation work therefore explores spatial relationships and boundary information.
- Most knowledge-distillation research focuses on classification, motivating investigation of efficient dense-prediction distillation beyond direct pixel-wise transfer.
3. Our Method
The method replaces strict or spatially uniform distillation with channel-wise alignment of normalized activation distributions, using asymmetric KL divergence to emphasize salient regions.
- Spatial Distillation: Existing dense-prediction KD methods align point-wise outputs or spatial relationships, including pair-wise, local-affinity, and class-prototype structures.
- Channel-wise Distillation: Channel-wise distillation converts each channel’s activations into a probability distribution before measuring teacher–student discrepancy.The transformation preserves spatial information while enabling probability-distance comparison.
- Channel-wise Distillation: The method uses asymmetric KL divergence between corresponding teacher and student channel distributions, with temperature controlling distribution softness and spatial coverage.Larger temperature values produce softer probabilities that focus on wider spatial regions.
- Channel-wise Distillation: Because teacher foreground probabilities receive greater attention, the student is guided toward similar activation distributions in salient regions while background mismatches matter less.The paper hypothesizes that this KL asymmetry benefits dense-prediction distillation.
4. Experiments
Experiments evaluate channel-wise distillation across semantic segmentation and object detection benchmarks, network structures, and ablations. The method consistently improves student performance while requiring less training computation than spatial distillation methods.
- Experimental Settings: Experiments use Cityscapes, ADE20K, Pascal VOC, and MS-COCO 2017 across semantic segmentation and object detection tasks.Cityscapes, ADE20K, and Pascal VOC evaluate segmentation; MS-COCO 2017 evaluates object detection.
- Semantic Segmentation: Our channel distillation method outperforms all spatial distillation methods, including the best spatial method AT by 2.5%.The comparison uses PSPNet-R101 as teacher and PSPNet-R18 as student on the Cityscapes validation set.
- Semantic Segmentation: The method improves class accuracy for traffic light, terrain, wall, truck, bus, and train, indicating effective transfer of structural knowledge.These class-level improvements are reported against PA and IFVD on Cityscapes.
- Ablation Study: The asymmetric KL divergence with normalized channel maps achieves the best performance among the evaluated ablation variants.The ablation compares variants using the same activation maps and training scheme.
- Ablation Study: The best configuration uses T = 4 and α = 3, while small T values reduce performance by focusing on limited salient pixels.A softer probability map may help distillation, and performance remains stable across a certain parameter range.
- Object Detection: On object detection, channel-wise distillation improves strong baseline student networks by about 3.4% mAP and performs better than previous state-of-the-art methods.RepPoint improves by 3.4%, compared with a 2% improvement reported for Zhang et al.; AP75 improves more significantly.
5. Conclusion
The paper proposes channel-wise distillation for dense prediction by converting each channel into a probability map and minimizing asymmetric KL divergence. Experiments show consistent improvements over state-of-the-art distillation methods across segmentation and detection benchmarks, while ablations support its efficiency and complementarity with spatial distillation.
- Conclusion: Channel-wise distillation normalizes each channel's activations into a probability map and minimizes asymmetric KL divergence between teacher and student networks.This differs from previous spatial distillation methods.
- Conclusion: The method consistently outperforms state-of-the-art distillation methods across four public benchmark datasets, network backbones, semantic segmentation, and object detection.The conclusion reports the result across both dense prediction task types.
- Conclusion: Ablation experiments demonstrate efficiency and effectiveness, and show that channel-wise distillation can complement spatial distillation methods.The authors suggest applying the method to other dense prediction tasks, including instance segmentation, depth estimation, and panoptic segmentation.
A. Results on Pascal VOC and ADE20K
Channel-wise distillation improves semantic segmentation across Pascal VOC and ADE20K student-network variants, with gains reported for different encoders and decoders.
- 3.83% improvement is reported for PSPNet-R18 on ADE20K, exceeding SKDS and IFVD by 1.51% and 1.21%, respectively.
- PSPNet-MBV2 reaches 27.97% with channel-wise distillation on ADE20K, surpassing the student, SKDS, and IFVD by 4.82%, 3.18%, and 2.64%.
- Deeplab-R18 improves from 66.81% to 69.97%, outperforming SKDS and IFVD by 1.84% and 1.55%, respectively.
- Deeplab-MBV2 increases from 50.80% to 54.62%, exceeding SKDS and IFVD by 2.51% and 1.23%, respectively.
B. More visualization results
Visualization results compare channel distribution distillation with attention transfer and inspect student channel distributions under three training paradigms.
- Figure 6 compares channel distribution distillation with attention transfer and reports better qualitative results for channel distribution distillation.
- Figure 7 visualizes student channel distributions for the original network, attention-transfer distillation, and channel distribution distillation.