Source-linked AI summary
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
Yongcheng Liu, Lu Sheng, Jing Shao, Junjie Yan, Shiming Xiang, Chunhong Pan
TL;DR
Multi-label classification needs to relate multiple labels to semantic regions, while existing region-based methods often rely on object-level annotations. This paper distills knowledge from an image-level-supervised WSD teacher into an MLIC student, and reports superior performance and efficiency on two large-scale datasets.
Problem
Multi-label image classification requires relating multiple classes to semantic regions and dependencies, while region-level cues often require laborious object-level annotations.
Source: §1 · pp. 1–2Method
The framework trains a WSD teacher from image-level annotations and distills its RoI features and whole-image predictions into an MLIC student.
Source: §1 · p. 2, §3.2 · p. 4Results
Experiments on MS-COCO and NUS-WIDE report superior performance and efficiency over state-of-the-art methods.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Multi-label image classification is a fundamental but challenging task towards general visual understanding. Existing methods found the region-level cues (e.g., features from RoIs) can facilitate multi-label classification. Nevertheless, such methods usually require laborious object-level annotations (i.e., object labels and bounding boxes) for effective learning of the object-level visual features. In this paper, we propose a novel and efficient deep framework to boost multi-label classification by distilling knowledge from weakly-supervised detection task without bounding box annotations. Specifically, given the image-level annotations, (1) we first develop a weakly-supervised detection (WSD) model, and then (2) construct an end-to-end multi-label image classification framework augmented by a knowledge distillation module that guides the classification model by the WSD model according to the class-level predictions for the whole image and the object-level visual features for object RoIs. The WSD model is the teacher model and the classification model is the student model. After this cross-task knowledge distillation, the performance of the classification model is significantly improved and the efficiency is maintained since the WSD model can be safely discarded in the test phase. Extensive experiments on two large-scale datasets (MS-COCO and NUS-WIDE) show that our framework achieves superior performances over the state-of-the-art methods on both performance and efficiency.
1 INTRODUCTION
Multi-label image classification must recognize multiple classes while relating them to semantic regions and class dependencies. The paper uses weakly supervised detection to guide a classification model through cross-task knowledge distillation.
1 INTRODUCTION
- MLIC requires understanding semantic regions and dependencies among multiple classes, beyond challenges shared with single-label classification.Source: §1 · p. 1
- Prior localization-based pipelines aggregate proposal scores but add redundant test-time computation, limiting their practicality for large-scale applications.Source: §1 · pp. 1–2
- WSD can locate class-specific semantic instances from image-level labels, and its predictions and regions can inform classification despite imperfect object boundaries.Source: Figure 1 · p. 1, §1 · p. 2
- The framework distills WSD knowledge into an MLIC student using image-level annotations, with WSD as teacher and classification as student.Source: §1 · p. 2
- Experiments on MS-COCO and NUS-WIDE report superior performance and efficiency over state-of-the-art methods.Source: §1 · p. 2
- The paper identifies cross-task knowledge distillation between WSD and MLIC as its first application of this approach.Source: §1 · p. 2
2 RELATED WORK
Prior MLIC work models label dependencies or locates semantic regions, often with added complexity or object-level supervision. The paper distinguishes its approach through WSD-to-MLIC distillation.
2 RELATED WORK
- MLIC methods model label dependencies with RNNs, graphs, or ensembles, with some requiring predefined relations and others adding complexity.Source: §2 · p. 2
- Region-localization approaches include bounding-box supervision, class-related attention maps, and scores on regions found by spatial transformers.Source: §2 · p. 2
- The reviewed methods may require predefined label relations or object-level annotations, while extra modules can reduce practicality.Source: §2 · p. 2
- Earlier distillation methods target the same task, whereas this framework transfers knowledge from WSD to MLIC through RoI features and whole-image predictions.Source: §2 · p. 2, Figure 2 · p. 3
3 METHODOLOGY
The methodology trains a weakly supervised detector from image-level annotations, then guides an end-to-end multi-label classifier through feature- and prediction-level knowledge transfer.
3 METHODOLOGY
- The classifier predicts each class by thresholding its image-level posterior probability.A class is included when its posterior exceeds its confidence threshold.Source: §3 · p. 3
3.1 Weakly-Supervised Detection (WSD) Model
- T-WDet uses EdgeBoxes proposals and RoI-pooled features, then fuses classification and detection branches to produce image-level class predictions and region detections.RoI pooling retains spatial information; the fused scores are summed across proposals for image-level predictions.Source: §3.1 · pp. 3–4
3.2 Knowledge Distillation from WSD to MLIC
- RoI-aware transfer uses WSD proposal confidences to select regions with NMS, then minimizes differences between teacher and student RoI features.The feature loss is applied on selected convolutional layers, with a possible transformation to match feature channels.Source: §3.2 · p. 4
- Class-aware prediction transfer matches softened teacher and student predictions while fitting hard labels, using learnable class-specific temperatures.The temperatures are learned during training rather than manually tuned.Source: §3.2 · pp. 4–5
3.3 Implementation Details
- Training freezes the trained detector while the student learns first from features and then from weighted prediction and label losses.The described implementation uses VGG16 and transfers features from conv5_3.Source: §3.3 · p. 5
- At test time, the student runs without the detector and adds no computation beyond a standard classification model.The student’s normal sigmoid outputs provide its final predictions.Source: §3.3 · p. 5
4 EXPERIMENT
Experiments on MS-COCO and NUS-WIDE compare the proposed framework with prior methods and examine its class-wise gains and component-level ablations. The reported results show improved classification after distillation and gains across different label types and class frequencies.
4.1 Datasets
- MS-COCO contains 80 object labels and NUS-WIDE 81 concept labels; both datasets are class-imbalanced, more severely on NUS-WIDE.The reported evaluation uses the official MS-COCO split and the split used in prior work for NUS-WIDE.Source: §4.1 · pp. 5–6
4.2 Evaluation Metrics and Compared Methods
- Evaluation reports macro and micro F1, top-3 F1, and mAP; the authors regard mAP as more informative because it measures ranking without a prediction threshold.Comparisons are restricted to VGG16-based methods without extra label information or ensemble testing; three advanced distillation methods are also evaluated.Source: §4.2 · pp. 6–7
4.3 Experimental Results
- After knowledge distillation with the proposed framework, S-Cls outperforms state-of-the-art methods on both MS-COCO and NUS-WIDE; on MS-COCO, it also beats advanced distillation methods and outperforms RMAM and RARLF with 224×224 input and a single-forward test.On MS-COCO, the framework uses only image-level annotations yet outperforms methods such as MIML-FCN-BB that require bounding-box annotations.Source: §4.3 · p. 7
- MS-COCO gains are relatively even across classes and focus on small objects, while NUS-WIDE gains concentrate on less frequent classes and span scenes, events, and objects.The authors relate the NUS-WIDE pattern to mitigating data imbalance and the MS-COCO pattern to semantic-region cues.Source: §4.3 · p. 7, Figure 4 · p. 8
4.4 Ablation Study
- 71.3 to 72.1 after adding class-aware distillation, and 72.3 to 73.8 with NMS applied to proposals by objectness confidence; confidence weighting improves performance again.The authors interpret objectness confidence as a reliable indicator of proposal semantics.Source: §4.4 · p. 7
- With supervised Faster R-CNN proposals, T-WDet improves from 78.6 to 81.1, while distilled S-Cls reaches 76.3 versus 74.6 with EB proposals.The authors describe the S-Cls gap between proposal settings as not obvious.Source: §4.4 · p. 8
5 CONCLUSION
The framework boosts multi-label classification by distilling knowledge from weakly-supervised detection using image-level annotations. Experiments on MS-COCO and NUS-WIDE show it outperforms state-of-the-art methods in performance and efficiency.
5 CONCLUSION
- On MS-COCO and NUS-WIDE, the framework outperforms state-of-the-art methods in both performance and efficiency.The WSD teacher can be discarded during testing.Source: §5 · p. 8
- The framework uses image-level annotations to distill WSD knowledge into a classification model through RoI feature-level and class-level prediction distillation.The WSD model is the teacher; the classification model is the student.Source: §5 · p. 8
- Future work will explore complementary cues that could facilitate weakly-supervised detection and further boost multi-label classification.Source: §5 · p. 8