Source-linked AI summary
Structured Knowledge Distillation for Dense Prediction
Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen
TL;DR
Dense prediction produces complex structured outputs and often requires large-capacity networks, while compact models are desirable for resource-limited devices. This paper transfers structure information through pair-wise and holistic distillation, improving compact networks across semantic segmentation, depth estimation, and object detection.
Problem
Dense prediction is more challenging than image-level prediction and often requires large-capacity networks, whereas compact models are needed for devices with limited computation resources.
Method
The paper distills structured knowledge using pair-wise affinity-graph alignment and holistic distillation, enforcing pair-wise and high-order consistency between compact and teacher networks.
Results
The proposed schemes improve recent compact networks across semantic segmentation, depth estimation, and object detection; for example, structured distillation improves C128-MV2 and C256-MV2 detection mAP by 0.9 and 0.8, respectively.
Takeaways & Limitations
Structured knowledge distillation is complementary to traditional pixel-wise distillation for compact dense prediction networks.
Abstract
from arXiv · showhide
In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at: https://git.io/StructKD
1 INTRODUCTION
Dense prediction requires compact models for resource-limited devices, but pixel-wise distillation overlooks relationships within structured outputs. This work transfers such structure through pair-wise and holistic distillation and evaluates it across three dense prediction tasks.
- Motivation: Dense prediction maps images to structured outputs for tasks including semantic segmentation, depth estimation, and object detection.These tasks assign labels or values across image locations and generally require large-capacity networks for high accuracy.
- Motivation: Compact dense prediction models are desirable for deployment on mobile and edge devices with limited computation resources.The paper contrasts lightweight architectures and model compression with the accuracy demands of dense prediction.
- Problem: Pixel-wise distillation transfers teacher outputs or feature embeddings independently at each pixel, neglecting important structure information.The paper identifies this as a limitation of directly adapting classification distillation to dense prediction.
- Approach: Pair-wise distillation aligns a static affinity graph capturing short- and long-range relations among locations in teacher and compact outputs.The graph is motivated by pair-wise Markov random fields and represents spatial similarities between locations.
- Approach: Holistic distillation uses adversarial training and a discriminator conditioned on the input image and output structures to align higher-order consistency.The compact network is encouraged to produce structures with embeddings similar to those of the teacher network.
- Evaluation: The approach improves compact networks across semantic segmentation, depth estimation, and object detection, with semantic-segmentation gains illustrated on Cityscapes.The reported experiments cover three dense prediction tasks, and the figure compares accuracy against computation and parameter cost.
2 APPROACH
The approach transfers structured knowledge from a teacher to a compact network for dense prediction through pixel-wise, pair-wise, and holistic distillation. Pair-wise distillation aligns affinity graphs, while holistic distillation uses adversarial training to match higher-order structure.
- Semantic segmentation: Semantic segmentation assigns one of C category labels to each pixel, producing a structured output from an RGB image.The network computes a feature map, applies a classifier, and upsamples the segmentation map to the input resolution.
- Pixel-wise distillation: Pixel-wise distillation aligns class probabilities independently at each pixel between the teacher and compact networks.This baseline treats segmentation as a collection of separate pixel-labeling problems.
- Pair-wise distillation: Pair-wise distillation aligns static affinity graphs that capture short- and long-range relations among spatial locations.Nodes aggregate local patches, while each node connects to its top-α spatial neighbors using Chebyshev distance.
- Pair-wise distillation: Pair-wise similarity distillation uses squared differences between teacher and student similarities, with connection range α and node granularity β controlling graph size.The similarity is computed from pooled node features, and fully connected graphs are avoided because of their high computational complexity.
- Holistic distillation: Holistic distillation matches higher-order structure by comparing embeddings of teacher and student outputs conditioned on the input image.A fully convolutional discriminator produces a holistic score, and adversarial training encourages student structures to receive teacher-like scores.
- Optimization: The overall objective combines multi-class cross-entropy with pixel-wise and structured distillation losses, while alternating discriminator and student updates.The compact network minimizes the objective, whereas the discriminator is optimized to score teacher outputs higher than student outputs.
- Extension to other tasks: The framework extends to object detection and depth estimation with task-specific output representations and minimal modifications.Depth can be treated as classification over discrete categories, while object detection predicts classes and a 4D bounding-box vector.
3 EXPERIMENTS
The experiments apply structured knowledge distillation first to semantic segmentation and then to object detection and depth estimation using strong baselines. These applications test the method across multiple structured output prediction tasks.
- Semantic segmentation: The semantic-segmentation experiments empirically verify the effectiveness of structured knowledge distillation and examine how it works.
- Other dense prediction tasks: Structured knowledge distillation is applied to object detection and depth estimation with fully convolutional frameworks and minimum modifications.
3.1 Semantic Segmentation
Semantic-segmentation experiments show that structured distillation improves compact students, with fully connected pair-wise graphs and capable discriminators providing stronger structure transfer.
- Ablation Study: Distillation improved student performance, with gains of 6.26%, 5.74%, and 2.9% for the evaluated ResNet configurations.The largest gains occurred for the smaller student and the model without ImageNet initialization; holistic distillation reached 62.7% mIoU for ResNet18 (0.5).
- Ablation Study: Fully connected affinity graphs improved pair-wise distillation, reaching around 71.37% mIoU, while 2 × 2 nodes offered a better efficiency–accuracy trade-off.Reducing node granularity can decrease connections while retaining stable correlations between locations.
- Ablation Study: Self-attention helped the discriminator capture structure, improving student accuracy on structured objects; deeper discriminators also helped adversarial training.The selected discriminator uses two self-attention blocks to balance performance, stability, and computational cost.
- Ablation Study: Holistic distillation produced higher discriminator scores and score distributions closer to the teacher, with self-attention and additional convolution blocks improving imitation.The discriminator score was used as an indicator of segmentation-map quality across teacher and student outputs.
- Segmentation Results: The approach improved segmentation across five compact networks, including 74.5 versus 71.9 validation quality compared with MD (Enhanced) on similar MobileNet models.For networks without pretraining, improvements were 7.3% for ResNet18 (0.5) and 6.6% for ESPNet-C, with lower model size in the MobileNet comparison.
- Segmentation Results: Structured distillation especially improved low-IoU categories and structured objects, including 17.23% for Bus and 10.03% for Truck.Qualitative Cityscapes results also showed more consistent labels for trunks, persons, buses, and traffic signs.
3.2 Depth Estimation
The depth-estimation experiments evaluate structured distillation on NYUD-V2 and find it more useful than pixel-level distillation, including when unlabeled data are added. The method also improves a strong Large-NYUD-V2 baseline.
- Evaluation: The NYUD-V2 evaluation reports relative error, log10 error, root mean squared error, and threshold accuracy metrics.The dataset contains 1,449 annotated indoor images, with 795 used for training.
- Ablation studies: Pixel-level distillation hardly improves depth-estimation accuracy, so the experiments use only structured knowledge distillation for this task.The paper attributes this difference from semantic segmentation to the teacher’s depth outputs being real-valued and often less accurate than ground-truth labels.
- Unlabeled data: Structured knowledge distillation outperforms pixel-wise distillation, and adding extra unlabeled data further improves depth-estimation accuracy.The comparison is reported in Table 10.
- Comparison with state-of-the-art: 13.0 relative error improves on the 13.5 baseline when structured distillation is added under the same Large-NYUD-V2 training setup.The baseline and distilled model use the constraints and training setup of the preceding method.
3.3 Object Detection
Object-detection experiments use FCOS with MobileNetV2 students and evaluate structured distillation on COCO. Structured distillation consistently outperforms pixel-wise MIMIC and improves compact detectors over their baselines.
- Implementation details: The experiments use a ResNeXt-32x8d-101-FPN FCOS teacher and c128-MNV2 or c256-MNV2 students.Distillation loss is applied at all output levels of the feature pyramid network.
- Evaluation: COCO evaluation reports mAP together with AP50, AP75, and scale-specific APs, APm, and APl.mAP averages AP across IoU values from 0.5 to 0.95 in steps of 0.05.
- Comparison of distillation methods: 0.9% mAP improvement from structured distillation exceeds the 0.4% improvement from pixel-wise MIMIC on the c256-MNV2 student.Both methods improve the detector relative to the baseline without distillation.
- Comparison of distillation methods: 32.1% mAP is achieved after combining the structured and pixel-wise distillation terms.The combined terms produce a further improvement over either distillation approach alone in the reported comparison.
- Results across student networks: 0.9 and 0.8 mAP improvements are obtained for C128-MV2 and C256-MV2, respectively, when structured and pixel-wise distillation are combined.These results compare students trained with and without distillation on COCO-minival.
- Results on test-dev: 33.9% mAP is obtained on minival after doubling training iterations with distillation, compared with 32.7% without distillation.The comparison uses the C128-MV2 student and the same doubled-iteration setting.
4 CONCLUSION
The paper concludes that structured knowledge distillation transfers dense-prediction structure through pair-wise and holistic schemes. Experiments across three tasks show effectiveness, and the methods complement traditional pixel-wise distillation.
- Conclusion: The paper presents pair-wise and holistic structured distillation schemes for training compact dense-prediction networks with teacher networks.The conclusion frames both schemes as ways to account for structure information in dense prediction.
- Conclusion: Experiments demonstrate effectiveness on semantic segmentation, depth estimation, and object detection.The evaluated models are described as recent compact networks.
- Conclusion: Structured knowledge distillation methods are complementary to traditional pixel-wise distillation methods.