Source-linked AI summary
Multi-Evidence Filtering and Fusion for Multi-Label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning
Weifeng Ge, Sibei Yang, Yizhou Yu
TL;DR
Image-level supervision alone makes accurate detection and segmentation difficult, while fully supervised methods require costly object- or pixel-level annotations. The paper proposes a weakly supervised curriculum pipeline that filters and fuses intermediate evidence before training task-specific networks. It reports state-of-the-art results for multi-label classification and weakly supervised detection, with competitive weakly supervised segmentation results across MS-COCO and PASCAL VOC datasets.
Problem
Image-level labels are cheaper than object- or pixel-level annotations, but weakly supervised methods struggle to predict accurate object locations, pixel labels, and multi-label outputs.
Method
The pipeline collects intermediate localization and pixel labels, filters and fuses object instances using metric learning and density-based clustering, and uses them to train task-specific networks fully supervised.
Results
The pipeline achieves state-of-the-art multi-label classification and weakly supervised object detection, and very competitive weakly supervised semantic segmentation across MS-COCO and PASCAL VOC.
Takeaways & Limitations
Filtering and fusing evidence from multiple techniques supports a single weakly supervised pipeline across recognition, detection, and segmentation tasks.
Takeaways & Limitations
The authors identify simplifying the pipeline's steps as an area requiring further investigation.
Abstract
from arXiv · showhide
Supervised object detection and semantic segmentation require object or even pixel level annotations. When there exist image level labels only, it is challenging for weakly supervised algorithms to achieve accurate predictions. The accuracy achieved by top weakly supervised algorithms is still significantly lower than their fully supervised counterparts. In this paper, we propose a novel weakly supervised curriculum learning pipeline for multi-label object recognition, detection and semantic segmentation. In this pipeline, we first obtain intermediate object localization and pixel labeling results for the training images, and then use such results to train task-specific deep networks in a fully supervised manner. The entire process consists of four stages, including object localization in the training images, filtering and fusing object instances, pixel labeling for the training images, and task-specific network training. To obtain clean object instances in the training images, we propose a novel algorithm for filtering, fusing and classifying object instances collected from multiple solution mechanisms. In this algorithm, we incorporate both metric learning and density-based clustering to filter detected object instances. Experiments show that our weakly supervised pipeline achieves state-of-the-art results in multi-label image classification as well as weakly supervised object detection and very competitive results in weakly supervised semantic segmentation on MS-COCO, PASCAL VOC 2007 and PASCAL VOC 2012.
1. Introduction
Weak supervision with image-level labels struggles to produce accurate object locations, pixel labels, and even multi-label predictions because annotations are incomplete. The paper addresses this with a curriculum pipeline that filters and fuses intermediate evidence before training task-specific networks.
- Image-level labels are less labor-intensive than object- or pixel-level annotations, but incomplete supervision makes accurate localization, pixel labeling, and multi-label prediction difficult.
- Top weakly supervised algorithms remain significantly less accurate than fully supervised counterparts, with lower precision from noisy outliers and lower recall from insufficient feature information.
- The four-stage process collects localization evidence, filters and fuses object instances, performs pixel labeling, and trains task-specific networks.
- The proposed curriculum pipeline obtains intermediate localization and pixelwise labels from image-level supervision, then trains detection, segmentation, and classification networks fully supervised.
- Metric learning and density-based clustering filter detected instances collected from multiple solution mechanisms to produce cleaner training instances.
- Image-level and object-level attention maps are fused with an object detection heat map to generate pixel-wise probability maps for fully convolutional pixel-labeling training.
2. Related Work
Related work frames weakly supervised detection and segmentation as learning from image-level labels through bottom-up, top-down, and hybrid mechanisms. The paper organizes these ideas into a curriculum that progresses from image-level to instance-level and pixel-level learning.
- Weakly supervised detection and segmentation locate and segment objects using image-level labels only, reducing reliance on data-hungry fully supervised learning.
- Bottom-up methods group pixels into object proposals and classify them, while top-down methods propagate neural-network outputs backward to obtain localization or pixel evidence.
- Neural attention provides a top-down mechanism for estimating pixel-wise class probabilities, but multiple labels can fuse category activations in shared neurons.
- The pipeline combines heatmaps, attention maps, metric learning, density-based clustering, and a single-label instance classifier across its image, instance, and pixel stages.
- The proposed pipeline decomposes pixel labeling into image-level, object-instance-level, and pixel-level learning tasks, each using current-stage data and previous-stage outputs.
3. Weakly Supervised Curriculum Learning
The curriculum pipeline decomposes weakly supervised pixel labeling into image-, instance-, and pixel-level stages, progressively filtering and fusing evidence before task-specific training.
- Overview: The pipeline decomposes pixel labeling into image-, instance-, and pixel-level stages within a unified curriculum-learning process.
- Image Level Stage: Object heatmaps from dense proposals and attention maps provide complementary localization evidence for generating object proposals.Heatmaps supply spatial priors but can extend beyond object boundaries, whereas attention maps may focus only on influential object parts.
- Image Level Stage: Attention proposals are retained when they cover more than 0.5 of heatmap proposals, then adjusted to lie between high- and low-confidence proposal regions.
- Instance Level Stage: Metric learning embeds object instances into a feature space, while density-based clustering removes instances dissimilar to normal examples within each class.Clustering selects a dense normal-instance cluster and treats instances outside it as outliers.
- Instance Level Stage: A single-label instance classifier re-labels proposals after filtering, helping recover recall lost through aggressive metric-learning and clustering screens.
- Pixel Level Stage: The pixel-level stage combines global and instance attention with object heatmaps, applying element-wise filtering and uncertainty-based thresholding to produce pixel labels.The final probability map is P = softmax(H ⊙ A), where H acts as a filter on the integrated attention map.
4. Object Recognition, Detection and Segmentation
The pipeline uses intermediate pixel labels, object instances, and attention maps to train task-specific networks for segmentation, detection, and multi-label classification.
- Semantic Segmentation: The semantic segmentation network is trained from generated pixel-wise labels, excluding pixels with uncertain class labels.At prediction time, atrous spatial pyramid pooling is used.
- Semantic Segmentation: Figure 4 illustrates pixel labeling and marks uncertain labels as white pixels in its final column.The figure documents the uncertainty-aware labeling process used before semantic segmentation training.
- Object Detection: Bounding boxes from connected pixels sharing semantic labels provide training instances for Fast R-CNN detection.These boxes are filtered with the object instance classifier because semantic label maps may contain noise.
- Multi-label Classification: A two-branch ResNet-101 network combines classification maps with normalized semantic-label attention maps for multi-label prediction.The segmentation branch supplies attention maps that weight important image regions in the classification branch.
5. Experimental Results
Experiments evaluate the pipeline on PASCAL VOC and MS-COCO across segmentation, detection, and multi-label classification. Results report strong detection and classification performance, while ablation supports the value of object instances and filtering.
- Implementation: The same hyper-parameter setting is used across datasets without validation-set tuning.Parameters were confirmed through visual checks on a few training segmentation samples.
- Qualitative Results: Figure 5 presents detection and semantic segmentation outputs for VOC 2012 and VOC 2007 test sets.Detection selects each class's highest-confidence proposal, while segmentation is post-processed with CRF.
- Evaluation: The experiments use image tags for training on PASCAL VOC and evaluate multi-label classification on the MS-COCO validation set.VOC detection uses mAP and CorLoc, while MS-COCO uses macro/micro precision, recall, and F1-measures.
- Object Detection: 47.5% mAP on PASCAL VOC 2012 test set is at least 5.0% higher than the latest state-of-the-art weakly supervised algorithms.On VOC 2007, the model reaches 51.2% mAP, 4.2% above the latest result, with CorLoc of 67% and 69.4% on the two trainval sets.
- Multi-Label Classification: The two-branch network achieves the highest reported F1-measures among state-of-the-art methods on MS-COCO.Compared with the baseline, P-C rises 6.6%, R-C falls 2.7%, and F1-C rises 2.1%.
- Ablation: Removing all object-instance processing decreases VOC 2007 detection mAP by 3.1%.The ablation removes both the instance-level stage and instance-attention operations in the pixel-level stage.
6. Conclusions
The paper presents a weakly supervised pipeline that fuses and filters object instances, performs uncertainty-aware pixel labeling, and uses the resulting labels for downstream tasks.
- Conclusions: The pipeline generates bounding boxes and attention maps from filtered object instances and pixelwise labels for detection and multi-label classification.It reports clearly better performance across object recognition, detection, and segmentation tasks.
- Conclusions: Simplifying the pipeline remains an open direction for further investigation.This conclusion identifies complexity as a remaining concern without specifying a particular simplification.