Source-linked AI summary
CNN: Single-label to Multi-label
Yunchao Wei, Wei Xia, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, Shuicheng Yan
TL;DR
Multi-label image classification is practical but difficult because images contain multiple, non-aligned objects and large-scale annotated training data is burdensome to obtain. HCP addresses this by applying a shared CNN to arbitrary object-segment hypotheses and max-pooling their outputs, achieving significant improvement over state-of-the-art methods on VOC 2007 and VOC 2012.
Problem
Multi-label image classification remains difficult because objects are not necessarily aligned, while collecting and annotating large-scale multi-label datasets is burdensome.
Method
HCP applies a shared CNN to an arbitrary number of object segment hypotheses and aggregates their outputs with cross-hypothesis max pooling.
Results
HCP achieved significant improvement over state-of-the-art methods in experiments on the Pascal VOC 2007 and VOC 2012 multi-label image datasets.
Takeaways & Limitations
HCP transfers CNNs pretrained on large-scale single-label datasets to multi-label classification without requiring bounding-box annotation during training.
Takeaways & Limitations
HCP's performance depends on extracted hypotheses covering all single objects, making high object-detection recall necessary.
Abstract
from arXiv · showhide
Convolutional Neural Network (CNN) has demonstrated promising performance in single-label image classification tasks. However, how CNN best copes with multi-label images still remains an open problem, mainly due to the complex underlying object layouts and insufficient multi-label training images. In this work, we propose a flexible deep CNN infrastructure, called Hypotheses-CNN-Pooling (HCP), where an arbitrary number of object segment hypotheses are taken as the inputs, then a shared CNN is connected with each hypothesis, and finally the CNN output results from different hypotheses are aggregated with max pooling to produce the ultimate multi-label predictions. Some unique characteristics of this flexible deep CNN infrastructure include: 1) no ground truth bounding box information is required for training; 2) the whole HCP infrastructure is robust to possibly noisy and/or redundant hypotheses; 3) no explicit hypothesis label is required; 4) the shared CNN may be well pre-trained with a large-scale single-label image dataset, e.g. ImageNet; and 5) it may naturally output multi-label prediction results. Experimental results on Pascal VOC2007 and VOC2012 multi-label image datasets well demonstrate the superiority of the proposed HCP infrastructure over other state-of-the-arts. In particular, the mAP reaches 84.2% by HCP only and 90.3% after the fusion with our complementary result in [47] based on hand-crafted features on the VOC2012 dataset, which significantly outperforms the state-of-the-arts with a large margin of more than 7%.
1 INTRODUCTION
HCP adapts CNNs to multi-label images by processing object-segment hypotheses with a shared CNN and max-pooling their outputs, avoiding bounding-box and hypothesis-label supervision. It also leverages single-label pretraining and produces intrinsic multi-label predictions.
- Motivation: Multi-label images challenge CNNs because objects are less aligned, may be partially visible or occluded, and require costly large-scale annotation.These conditions undermine assumptions that support single-label CNN classification and make comprehensive multi-label training data difficult to obtain.
- HCP framework: HCP processes an arbitrary number of object-segment hypotheses through a shared CNN and aggregates their outputs with cross-hypothesis max pooling.The pooled output is a c-dimensional vector of final multi-label predictions.
- Training requirements: HCP requires no ground-truth bounding-box information for multi-label training, reducing annotation burden compared with methods that use such supervision.The paper also states that this supports transfer to new multi-label datasets.
- Robustness: Max pooling suppresses noisy hypotheses by retaining high predictive scores from hypotheses containing objects and also addresses redundant hypotheses.The method assumes that at least one hypothesis contains the object of interest.
- Training requirements: HCP needs no explicit hypothesis labels and can initialize its shared CNN through pretraining on a large-scale single-label dataset such as ImageNet.The shared CNN is subsequently fine-tuned on the target multi-label dataset.
- Outputs and evaluation: HCP outputs normalized label probabilities whose values serve intrinsically as classification confidence scores for the corresponding categories.The paper reports extensive evaluation on Pascal VOC 2007 and VOC 2012 against state-of-the-art methods.
2 RELATED WORK
Earlier multi-label approaches used bag-of-words or deep-learning frameworks, while transfer learning and proposal-based CNN methods addressed limited data and object localization. HCP removes bounding-box and hypothesis-label requirements while using fewer selected hypotheses to speed training.
- Existing frameworks: Multi-label image classification methods generally follow bag-of-words or deep-learning frameworks.Bag-of-words systems combine hand-crafted representation, classification, and context modelling modules.
- Deep learning models: Deep CNNs have shown strong image-classification ability on single-label datasets such as CIFAR-10/100 and ImageNet.Their success motivated applying CNN architectures to multi-label problems.
- Deep learning models: CNNs pre-trained on diverse large datasets can transfer to image datasets with insufficient training data, but global CNN features are not optimal for less-aligned multi-label objects.Multi-label objects may also exhibit partial visibility and occlusion.
- Proposal-based methods: Proposal-based multi-label methods improved performance but highly depend on ground-truth bounding boxes, limiting transfer to datasets without that information.HCP is presented as avoiding this dependency while remaining robust to noisy and redundant hypotheses.
3 HYPOTHESES-CNN-POOLING
HCP combines selected object hypotheses, a shared CNN, and cross-hypothesis max pooling to produce multi-label predictions without requiring explicit hypothesis labels or ground-truth boxes. Its design uses efficient hypothesis selection, ImageNet pre-training with image fine-tuning, and pooling to handle limited data and noisy hypotheses.
- 3.1 Hypotheses Extraction: HCP selects a smaller set of object hypotheses from BING proposals and feeds them into a shared CNN whose confidence vectors are fused for prediction.The proposed HS method clusters proposals, filters unsuitable boxes, and selects top-scoring hypotheses from each cluster.
- 3.1 Hypotheses Extraction: The HS method seeks high object recall with few hypotheses, because all hypotheses are processed simultaneously and larger sets increase computation and hardware demands.BING offers high recall but its 1,000 hypotheses remain too numerous for HCP, motivating further selection.
- 3.2 Initialization of HCP: HCP initializes the shared CNN by pre-training on ImageNet and fine-tuning on the target multi-label dataset using the entire image as input.ImageNet parameters are transferred except for the final fully connected layer, whose category count differs for Pascal VOC.
- 3.2 Initialization of HCP: Image-fine-tuning is critical: assigning different learning rates across layers preserves low-level pre-trained representations while adapting later layers to the target dataset.The authors report that removing image fine-tuning caused VOC 2007 performance to drop dramatically.
- 3.3 Hypotheses-fine-tuning: Cross-hypothesis max pooling preserves high responses from hypotheses containing objects while ignoring noisy hypotheses, yielding an integrative multi-label prediction.For a test image, hypotheses produce category responses independently, and pooling retains labels such as car, horse, and person.
4 EXPERIMENTAL RESULTS
Experiments on VOC 2007 and VOC 2012 evaluate HCP against hand-crafted- and CNN-based state-of-the-art methods. HCP improves performance without ground-truth bounding-box annotation, while fusion with complementary hand-crafted features further enhances VOC 2012 results.
- Datasets and competing methods: Experiments evaluate HCP on VOC 2007 and VOC 2012 against state-of-the-art approaches based on hand-crafted and learned features.The compared methods are divided into hand-crafted-feature and learned-feature approaches.
- VOC 2007 results: 74.4% mAP is achieved by the image-fine-tuned model on VOC 2007, outperforming CNN features with an SVM classifier.
- Prediction analysis: HCP can recover a missed car category because its hypothesis-based processing treats foreground and background objects equivalently, unlike whole-image fine-tuning.
- VOC 2012 results: 81.7% versus 78.7% shows HCP-1000C outperforming PRE-1000C on VOC 2012, while HCP-2000C improves 1.4% over PRE-1512.HCP-1000C also improves 2.7% over the result reported by LeCun et al. without bounding-box annotation.
- Fusion and category analysis: Late fusion reaches 90.3% mAP on VOC 2012, demonstrating complementarity between HCP’s deep-network predictions and hand-crafted-feature results.The comparison also reports stronger hand-crafted-feature performance for rigid categories and stronger CNN-based performance for articulated categories.
5 CONCLUSIONS
HCP addresses multi-label image classification by transferring CNNs pretrained on single-label datasets without requiring bounding-box annotations. Evaluations on VOC 2007 and VOC 2012 found significant improvement over state-of-the-art methods, with further gains from late fusion with hand-crafted features.
- Contributions: HCP transfers CNNs pretrained on large-scale single-label datasets, such as ImageNet, to multi-label image classification.The framework requires no bounding-box annotation for training, facilitating adaptation to new multi-label datasets.
- Evaluation: HCP produced significant improvement over state-of-the-art methods on the VOC 2007 and VOC 2012 datasets.
- Fusion: Late fusion of CNN outputs with hand-crafted feature schemes further enhanced classification performance.