Source-linked AI summary
Deep Convolutional Ranking for Multilabel Image Annotation
Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, Sergey Ioffe
TL;DR
Multilabel image annotation requires assigning several labels to each image, but conventional objectives and features do not directly target top-k annotation accuracy. The paper trains convolutional networks with ranking-based losses, emphasizing weighted approximate ranking, and reports about a 10% improvement over conventional visual features on NUS-WIDE. Its scope includes limitations of pairwise ranking for top-k accuracy and a looser upper bound in the approximate objective.
Problem
Multilabel annotation is challenging because images may have multiple labels, while pairwise ranking optimizes AUC rather than directly optimizing top-k annotation accuracy.
Method
The paper trains deep convolutional neural networks for multilabel annotation using and comparing ranking-based losses, including weighted approximate ranking.
Results
About 10%: CNN+Softmax outperforms the VisualFeature+SVM baseline on NUS-WIDE.
Takeaways & Limitations
Weighted approximated-ranking loss works particularly well for multilabel annotation, supporting top-k ranking as an effective way to train the network.
Takeaways & Limitations
Pairwise ranking does not directly optimize top-k annotation accuracy, and the approximate objective is a looser upper bound than the original WARP loss.
Abstract
from arXiv · showhide
Multilabel image annotation is one of the most important challenges in computer vision with many real-world applications. While existing work usually use conventional visual features for multilabel annotation, features based on Deep Neural Networks have shown potential to significantly boost performance. In this work, we propose to leverage the advantage of such features and analyze key components that lead to better performances. Specifically, we show that a significant performance gain could be obtained by combining convolutional architectures with approximate top-$k$ ranking objectives, as thye naturally fit the multilabel tagging problem. Our experiments on the NUS-WIDE dataset outperforms the conventional visual features by about 10%, obtaining the best reported performance in the literature.
1 Introduction
Multilabel image annotation addresses the practical need to assign multiple semantic tags to images, while this work investigates deep convolutional networks and ranking-based objectives for the task.
- Motivation: Multilabel annotation assigns multiple semantic tags to one image, reflecting descriptions of objects, activities, and scenes in real-world imagery.Images from Flickr and the Internet commonly carry several tags or textual descriptions rather than a single class label.
- Background: Most prior vision work focused on single-label classification and conventional visual features, although convolutional neural networks had shown strong results for image recognition.The related work discusses features such as sparse coding, Fisher vectors, VLAD, and spatial pyramid matching, alongside CNN advances.
- Approach: The paper leverages expressive convolutional networks for multilabel annotation and studies components that improve performance.Its architecture uses convolutional and densely connected layers as the basic network structure.
- Approach: The authors compare multilabel losses, including AUC-optimizing ranking loss and Tagprop's cross-entropy loss, while proposing top-k ranking loss for network training.The comparison targets objectives suited to multilabel tagging and reports experiments on NUS-WIDE.
- Results: The experiments report a significant performance boost over conventional visual features and the best retrieval performance on NUS-WIDE at the time.The evaluation uses the largest publicly available multilabel dataset described in the paper.
2 Multilabel Deep Convolutional Ranking Net
The method uses a convolutional network for multilabel annotation and compares loss functions tailored to ranking multiple labels. It emphasizes top-k optimization through a weighted approximate ranking objective while noting limitations of pairwise ranking and the resulting approximation.
- Network and losses: The approach adopts a convolutional architecture and focuses on loss functions designed for multilabel prediction.The network uses five convolutional and three densely connected layers, with image resizing and patch extraction for augmentation.
- Softmax loss: The softmax loss converts network activations into class probabilities and minimizes KL divergence against normalized multilabel targets.For an image with multiple labels, the binary label vector is normalized by its L1 norm to form ground-truth probabilities.
- Pairwise ranking: The pairwise-ranking loss encourages positive labels to score above negative labels but optimizes AUC rather than top-k annotation accuracy.Because the task prioritizes top-k annotations, the paper identifies this mismatch as a limitation of the pairwise objective.
- Weighted approximate ranking: The WARP loss uses stochastic sampling to optimize top-k annotation accuracy within the deep network’s stochastic optimization framework.Negative labels are sampled until a violation is found, and the estimated rank determines the loss contribution.
- Weighted approximate ranking: The weighting function assigns smaller penalties to highly ranked positive labels and larger penalties to positive labels that are ranked lower.The method sets α_i = 1/j, while the rank is estimated from the number of negative-label sampling trials required to find a violation.
- Weighted approximate ranking: The optimized approximate objective is a looser upper bound than the original WARP loss because other positive labels may rank above a given positive label.In the multilabel setting, the probability of sampling a violator is bounded by the original rank-based expression rather than necessarily equal to it.
3 Visual Feature based Image Annotation Baslines
The paper establishes strong visual-feature baselines for multilabel annotation using handcrafted descriptors with kNN and one-vs-all SVM classifiers.
- Visual Features: Nine visual features are combined into a strong baseline representation for image annotation.The combined descriptor has 36,472 dimensions before dimensionality reduction.
- Visual Features: GIST features are extracted from resized images using three scales across RGB channels, producing 960-dimensional vectors.
- Visual Features: SIFT variants use dense and Harris sampling, three local descriptors, a 1,000-word codebook, and a two-level spatial pyramid.The resulting vectors are 5,000-dimensional, with six named variants.
- Visual Features: HOG features are quantized with a 1,000-size codebook and spatial pyramid, yielding 5,000-dimensional vectors.
- Classifiers: Weighted kNN ranks tags from neighboring training images, while linear SVMs rank outputs from separate one-vs-all tag classifiers.kNN is used because the annotation problem is highly nonlinear and has a heavily tailed label distribution.
4 Experiments
Experiments evaluate multilabel annotation on NUS-WIDE using top-k predictions and multiple recall and precision measures. The CNN substantially improves over visual-feature baselines, with WARP especially effective for infrequent tags.
- Dataset: NUS-WIDE contains 269,648 Flickr images, 81 tags, and typically 2–5 tags per image; 150,000 images are used for training.After excluding unannotated images, 209,347 images remain for training and testing.
- Evaluation Protocols: Evaluation assigns k highest-ranked tags per image and reports mean-per-class recall and precision for k = 3 and k = 5.Per-tag scores compare predicted tags with ground-truth annotations.
- Evaluation Protocols: Overall recall and precision are added because mean-per-class metrics can be biased toward infrequent tags.Overall metrics instead give frequent classes greater influence on final performance.
- Evaluation Protocols: An upper-bound protocol samples or supplements ground-truth tags to produce k labels per image when images have varying numbers of annotations.The resulting baseline represents the best possible performance when ground truth is known.
- Results: About 10%: CNN+Softmax outperforms the VisualFeature+SVM baseline for three predicted tags per image.WARP improves substantially over softmax, whereas pairwise ranking does not improve softmax; results for k = 5 show similar trends.
- Results: WARP performs better than other losses for infrequent classes, while different losses perform comparably for frequent classes.Per-class recall and precision are sorted by tag frequency in the corresponding analyses.
5 Discussion and Future Work
The paper concludes that ranking-based training is effective for deep convolutional networks in multilabel annotation. It identifies WARP as particularly effective and proposes future training on large collections of noisy Internet images.
- Discussion: The study trains deep convolutional networks for multilabel annotation with ranking-based loss functions.
- Discussion: WARP works particularly well for multilabel annotation and demonstrates the effectiveness of top-k ranking on NUS-WIDE.
- Future Work: Future work will use very large collections of noisy-labeled multilabel images from Flickr or image searches.
- Discussion: Qualitative WARP annotations can remain meaningful even when some predicted tags do not match the ground truth.