Source-linked AI summary
Cross-Image Relational Knowledge Distillation for Semantic Segmentation
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, Qian Zhang
TL;DR
Existing segmentation KD methods often use information from individual samples and therefore miss global pixel relations across images. CIRKD transfers cross-image pixel-to-pixel and pixel-to-region relations with memory-based embeddings, and experiments show strong performance across multiple datasets and methods.
Problem
Existing segmentation KD methods can ignore structured pixel context and global pixel dependencies across images when guiding compact students.
Method
CIRKD transfers teacher-generated cross-image pixel-to-pixel and pixel-to-region similarity relations using memory-bank queues.
Results
CIRKD achieves the best reported distillation performance across Cityscapes, CamVid and Pascal VOC experiments and outperforms competing methods.
Takeaways & Limitations
Cross-image relational distillation guides students to learn broader pixel dependencies and improves semantic segmentation performance.
Abstract
from arXiv · showhide
Current Knowledge Distillation (KD) methods for semantic segmentation often guide the student to mimic the teacher's structured information generated from individual data samples. However, they ignore the global semantic relations among pixels across various images that are valuable for KD. This paper proposes a novel Cross-Image Relational KD (CIRKD), which focuses on transferring structured pixel-to-pixel and pixel-to-region relations among the whole images. The motivation is that a good teacher network could construct a well-structured feature space in terms of global pixel dependencies. CIRKD makes the student mimic better structured semantic relations from the teacher, thus improving the segmentation performance. Experimental results over Cityscapes, CamVid and Pascal VOC datasets demonstrate the effectiveness of our proposed approach against state-of-the-art distillation methods. The code is available at https://github.com/winycg/CIRKD.
1. Introduction
Semantic segmentation requires dense, structured pixel predictions, while compact networks need knowledge distillation to reduce deployment costs. CIRKD transfers global cross-image pixel relations through pixel-to-pixel and pixel-to-region distillation, improving student segmentation.
- Semantic segmentation produces dense pixel labels but high-performing networks can be computationally costly for resource-limited mobile devices.
- Classification-oriented KD may impose negative constraints by strictly aligning coarse feature maps and ignoring structured pixel context.
- CIRKD transfers global pixel relationships across training images using pixel-to-pixel and pixel-to-region distillation.Pixel-to-region relations compare pixels with class-wise prototypes formed by averaging same-class pixel embeddings.
- A pixel and region queue in a teacher-generated memory bank provides abundant, consistent embeddings for modelling long-range relations beyond small mini-batches.Teacher embeddings are generated by the pre-trained, frozen teacher, while current teacher and student embeddings serve as anchors.
- CIRKD outperforms state-of-the-art distillation methods on Cityscapes, CamVid and Pascal VOC benchmarks.The experiments evaluate DeepLabV3 and PSPNet architectures across the three datasets.
2. Related Work
Related work develops efficient segmentation networks, conventional knowledge-distillation categories, and specialized methods for transferring structured pixel context. These approaches motivate CIRKD's focus on cross-image relations.
- Semantic Segmentation: Semantic segmentation research captures long-range context through architectures such as atrous convolutions and multiscale feature processing.
- Semantic Segmentation: Efficient segmentation networks use lightweight operations and multibranch or cascade designs to balance computational cost with segmentation quality.
- Knowledge Distillation: Knowledge distillation transfers knowledge from a cumbersome teacher into a smaller, faster student through probability-based, feature-based, or relation-based approaches.
- Knowledge Distillation: Segmentation-specific KD methods distill local affinity, global pairwise relations, intra-class variation, or channel-wise category masks.
3. Methodology
CIRKD distills cross-image semantic relations by aligning student and teacher pixel similarities, using memory queues to model broader pixel and region dependencies.
- Motivation: Semantic segmentation distillation aligns dense pixel predictions, while conventional pixel-wise KD neglects semantic relations between pixels.Prior segmentation KD methods model relations within a single image, but not dependencies across global images.
- Overview: CIRKD constructs cross-image pixel relations and transfers the teacher’s structured pixel embedding relationships to the student.The method includes both pixel-to-pixel and pixel-to-region distillation.
- Mini-batch-based Pixel-to-Pixel Distillation: Mini-batch pixel-to-pixel distillation computes cross-image pair-wise similarity matrices and aligns row-wise student and teacher distributions with KL divergence.Pixel embeddings are l2-normalized, and softmax normalization with temperature τ removes magnitude gaps before alignment.
- Memory-based Pixel-to-Pixel Distillation: Small segmentation batches limit global dependency modeling, so CIRKD uses a class-aware FIFO pixel queue containing sampled teacher embeddings.The queue samples V pixel embeddings per class, with V ≪ Np, to reduce redundancy and preserve diversity.
- Memory-based Pixel-to-Pixel Distillation: Memory-based pixel-to-pixel distillation makes student-anchor similarities to queued pixel embeddings mimic the teacher’s normalized similarity matrix.Teacher embeddings are stored because the frozen teacher provides consistent features during distillation.
- Memory-based Pixel-to-Region Distillation: Pixel-to-region distillation averages class-specific pixel embeddings into region embeddings and aligns student-teacher pixel-to-region similarity matrices.A class-balanced region queue stores representative teacher region embeddings for cross-image relations.
4. Experiments
Experiments across Cityscapes, CamVid, and Pascal VOC evaluate CIRKD with multiple student architectures and structured distillation baselines. CIRKD consistently achieves strong segmentation results, while ablations support the value of memory-based cross-image relations and both pixel-to-pixel and pixel-to-region losses.
- Experimental Setup: Experiments use Cityscapes, CamVid, and Pascal VOC with DeepLabV3-ResNet101 as teacher and DeepLabV3 or PSPNet students using ResNet-18 or MobileNetV2 backbones.Evaluation follows single-scale inference, with comparisons against SKD, IFVD, and CWD using the same pretrained teacher.
- Cityscapes: 0.60% validation mIoU and 0.78% test mIoU gains over CWD are achieved on average across four Cityscapes student networks.CIRKD obtains the best performance across students with similar or different architecture styles.
- Cityscapes: 10.4% and 9.4% relative improvements on Wall are obtained over the baseline and CWD, respectively, while qualitative outputs show more consistent semantic labels with ground truth.The class-wise gains are especially pronounced for categories with low IoU scores.
- CamVid: 0.50% and 0.73% mIoU gains over CWD are obtained with DeepLabV3 and PSPNet, respectively, on CamVid.CIRKD achieves the best performance consistently among the evaluated distillation methods.
- Pascal VOC: 0.48% and 0.79% mIoU improvements over CWD are obtained with DeepLabV3 and PSPNet, respectively, on Pascal VOC.The results support CIRKD's scalability to visual object segmentation.
- Ablation Study: Applying all distillation losses reduces the student-teacher performance gap from 4.95% to 2.65%.Memory-based pixel-to-pixel distillation outperforms its mini-batch counterpart, and pixel-to-pixel distillation is more informative than pixel-to-region distillation in the ablation.
- Parameter Analysis: Distillation performance increases with larger pixel and region queues and with larger Kp and Kr, but eventually saturates; τ = 0.1 is reported as the best temperature.Performance saturation is reported at Kp = 4096 for pixel-to-pixel and Kr = 1024 for pixel-to-region distillation.
5. Conclusion
CIRKD transfers global pixel correlations from teacher to student networks for semantic segmentation, helping students learn broader pixel dependencies.
- CIRKD transfers global pixel correlations from the teacher to the student for semantic segmentation.
6. Appendix
Figure 8 provides an overview of mini-batch-based pixel-to-pixel distillation.
- Figure 8 presents the overview of mini-batch-based pixel-to-pixel distillation.