Source-linked AI summary
Knowledge Adaptation for Efficient Semantic Segmentation
Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, Youliang Yan
TL;DR
Semantic segmentation must balance dense-estimation accuracy against the computation required by high-resolution feature maps. The paper proposes knowledge distillation using a transferred latent domain and affinity-based long-range dependency modeling, improving compact students while reducing computation.
Problem
High-resolution FCNs preserve detailed semantic information but require heavy computation, whereas subsampling improves efficiency at the cost of estimation accuracy.
Method
The method distills teacher knowledge through an autoencoder-derived latent representation and an affinity module that models non-local interactions across image positions.
Results
The method improves student performance without extra parameters or computations and achieves comparable or better results with only 8% FLOPS of a large-resolution model.
Takeaways & Limitations
Compact semantic-segmentation models can retain or improve performance while using substantially less computation than models with larger-resolution outputs.
Takeaways & Limitations
Knowledge distillation methods developed for image classification may not transfer directly to semantic segmentation because teacher and student decision spaces can differ.
Abstract
from arXiv · showhide
Both accuracy and efficiency are of significant importance to the task of semantic segmentation. Existing deep FCNs suffer from heavy computations due to a series of high-resolution feature maps for preserving the detailed knowledge in dense estimation. Although reducing the feature map resolution (i.e., applying a large overall stride) via subsampling operations (e.g., pooling and convolution striding) can instantly increase the efficiency, it dramatically decreases the estimation accuracy. To tackle this dilemma, we propose a knowledge distillation method tailored for semantic segmentation to improve the performance of the compact FCNs with large overall stride. To handle the inconsistency between the features of the student and teacher network, we optimize the feature similarity in a transferred latent domain formulated by utilizing a pre-trained autoencoder. Moreover, an affinity distillation module is proposed to capture the long-range dependency by calculating the non-local interactions across the whole image. To validate the effectiveness of our proposed method, extensive experiments have been conducted on three popular benchmarks: Pascal VOC, Cityscapes and Pascal Context. Built upon a highly competitive baseline, our proposed method can improve the performance of a student network by 2.5\% (mIOU boosts from 70.2 to 72.7 on the cityscapes test set) and can train a better compact model with only 8\% float operations (FLOPS) of a model that achieves comparable performances.
1. Introduction
Semantic segmentation must balance dense-detail accuracy against computational efficiency. The paper addresses this dilemma with segmentation-specific knowledge distillation that adapts teacher knowledge into a compact student and adds affinity-based relational supervision.
- Motivation: High-resolution feature maps preserve detailed semantic structures but impose heavy computations and limit semantic segmentation practicability.Deep FCN methods often use a small overall stride for dense estimation.
- Motivation: Reducing feature-map resolution through pooling or convolution striding improves efficiency but causes unsatisfactory accuracy by losing detailed information.
- Motivation: Existing knowledge-distillation methods largely target image-level classification and do not directly address spatial context or teacher–student feature mismatches in segmentation.Teacher and student feature maps can have inconsistent context and mismatched features.
- Proposed approach: The proposed method distills teacher knowledge into a compact latent representation learned through an autoencoder, rather than forcing direct output imitation.The compact representation is intended to remove redundant and noisy knowledge while accommodating architectural differences.
- Contributions and validation: The affinity distillation module captures long-term dependencies, while experiments report large student improvements without extra parameters or computations and comparable performance at 8% FLOPS.
2. Related Work
Prior work established strong segmentation methods and knowledge distillation, but segmentation-specific distillation remains challenging because architectures differ in spatial and long-range context. The paper’s approach is motivated by compressing dense representations while retaining useful contextual information.
- Semantic Segmentation: Semantic segmentation progress has relied on rich spatial information and sizable receptive fields.
- Semantic Segmentation: DeepLab-CRF, CRF-RNN, and RefineNet improve segmentation through boundary refinement, embedded CRF learning, or high-resolution outputs with long-range residual modules.
- Semantic Segmentation: Small overall output strides capture detailed information better but substantially increase computation and storage in deep networks.The discussion notes that smaller output strides can affect many layers or residual blocks.
- Knowledge Distillation: Knowledge distillation transfers teacher soft outputs or intermediate representations, but direct feature alignment can overlook differences in resolution, channels, architecture, and abstraction capability.
- Knowledge Distillation: Semantic-segmentation distillation is limited by differing context and long-range dependency capabilities between teacher and student networks, and by temperature sensitivity on large benchmarks.
3. Proposed Method
The method distills a frozen teacher’s high-resolution semantic knowledge into a compact student through a learned latent representation and affinity-based long-range information. The student is trained with ground-truth supervision alongside adaptation and affinity-transfer objectives.
- Knowledge Translation and Adaptation: A frozen teacher produces high-resolution features, while a lower-resolution student is trained for faster inference using compressed knowledge and affinity information.The framework targets students with smaller feature maps, such as a 16s-output-stride student taught by an 8s teacher.
- Knowledge Translation and Adaptation: An auto-encoder compresses teacher features into a latent format that is easier for the student to learn despite structural differences between networks.The auto-encoder is trained to reconstruct teacher features, with encoder and decoder components defining the translated representation.
- Affinity Distillation Module: The affinity module computes non-local interactions between feature-map positions and transfers the teacher’s long-range dependencies to the student.Affinity matrices are matched with an ℓ2 loss after teacher translation and student adaptation.
- Affinity Distillation Module: Visualization shows that affinity distillation changes the student from highlighting mainly local similar patterns to capturing long-range or global information for more robust decisions.Figure 3 compares the input point, the student without affinity distillation, and the enhanced affinity map.
- Training Process: The student is optimized with cross-entropy, latent adaptation, and affinity-transfer losses, while the teacher remains pretrained and frozen.The three losses are combined using weights β and γ during student training; the auto-encoder is trained beforehand.
4. Experiments
Experiments evaluate the method through ablations, alternative teacher and student networks, comparisons with KD and FitNet, and tests on Pascal VOC, Pascal Context, and Cityscapes. Across these settings, knowledge adaptation and affinity distillation improve compact segmentation models while preserving computational efficiency.
- Experimental settings: Experiments cover Pascal VOC, Cityscapes, and Pascal Context, with performance evaluated using mIOU.Pascal VOC uses pixel intersection-over-union averaged across 21 classes; Cityscapes evaluates 19 semantic labels, while Pascal Context evaluates 59 frequent classes plus background.
- Ablation study: The affinity distillation module raises mIOU from 70.57 to 71.53, while knowledge adaptation adds another 0.97 point on Pascal VOC.These ablations use a single scale on Pascal VOC validation data without COCO pretraining.
- Ablation study: A MobileNetV2 student with output stride 16 outperforms the output-stride-8 version using only 31% FLOPS.With further comparisons, the 16s model performs better than a baseline with 4s output using only 8% FLOPS and no extra parameters.
- Different networks: Guidance from ResNet-50 and Xception-41 improves MobileNetV2-1.0 mIOU by 1.93 and 1.83 points, respectively.MobileNetV2-1.3 and MobileNetV2-1.4 also improve by 1.66 and 0.71 points under the tested settings.
- Comparison with distillation methods: The proposed method outperforms KD across tested hyper-parameters and datasets and exceeds FitNet by 1.2 points.The reported loss curve indicates that translating knowledge into a learnable format improves transfer from teacher to student.
- Benchmark results: The method improves mIOU by 1.3 points on Pascal Context and reaches 75.8 mIOU on Pascal VOC.On Pascal VOC, it also improves the MobileNetV2 baseline by about 1 point.
- Benchmark results: On Cityscapes, the method achieves 70.3 mIOU on validation and 72.7 mIOU on testing, improving the baseline by 2.1 and 2.5 points without extra parameters or computation overhead.The experiments report these results on both validation and test datasets.
5. Conclusion
The paper concludes that its semantic-segmentation distillation framework improves compact student models by translating high-level teacher features into a learnable compact format. It reports substantial performance gains without extra parameters or computation and better results with much lower computational overhead.
- Conclusion: The framework translates high-level teacher features into a compact format that is easier for the student model to learn.The conclusion presents this translation as the mechanism for improving student-model performance.
- Conclusion: The method improves student performance without introducing extra parameters or computations.This claim is stated as a principal conclusion of the experiments.
- Conclusion: The method achieves better results with much less computation overhead.The conclusion summarizes the efficiency-performance trade-off demonstrated by the experiments.
6. Implementation details
Implementation uses convolutional translator and adapter modules, dataset-specific training schedules, and fixed teacher and autoencoder parameters during student training. The reported configurations specify architecture, optimization, and evaluation procedures.
- Knowledge Translator: The knowledge translator uses three convolutional and three transposed-convolutional layers with 3 × 3 kernels, batch normalization, and ReLU activations.The first encoder convolution and last decoder convolution use stride 2; all six layers match the teacher’s final-feature channel count.
- Knowledge Adapter: The knowledge adapter applies three stride-1 convolutional layers while preserving spatial resolution and matching the teacher’s final-feature channel count.Each layer uses a 3×3 kernel, padding 1, batch normalization, and ReLU.
- Training Process: Pascal VOC training uses COCO pretraining followed by trainaug training, whereas Pascal Context and Cityscapes use their specified training splits and iteration schedules.Cityscapes training includes fine-tuning on trainval and train-coarse data before test evaluation.
7. Results Visualization
Visual comparisons on Pascal VOC and Cityscapes show that the proposed method produces more accurate segmentation results than the MobileNetV2 student network, with Pascal VOC results also surpassing a prior knowledge-distillation method.
- On Pascal VOC, the proposed method generates more accurate segmentation results than the MobileNetV2 student network.The comparison uses a strong MobileNetV2 baseline without ASPP or a decoder.
- On Pascal VOC, the proposed method also outperforms the results of the prior knowledge-distillation method [11].
- Figure 7 compares input images, ground truth, MobileNetV2 student outputs, prior distillation outputs, proposed-method outputs, and ResNet50 teacher outputs.
- Figure 8 presents Cityscapes examples comparing input images, ground truth, student-network results, and proposed-method results.