Source-linked AI summary
Fast Zero-Shot Image Tagging
Yang Zhang, Boqing Gong, Mubarak Shah
TL;DR
It is unclear whether linear word-vector offsets capture visual relevance among tags for an image. Fast0Tag estimates an image-specific principal direction from the image, and relevant-tag vectors empirically rank ahead of irrelevant ones along such a direction.
Problem
It is unclear whether linear word-vector offsets capture visual relevance relations that partition an image’s tags into relevant and irrelevant words.
Source: §1 · p. 1, §3.1 · p. 3Method
Fast0Tag estimates an image-specific principal direction from the input image using linear mappings or deep neural networks, then ranks tag vectors along it.
Source: §1 · p. 2Results
Takeaways & Limitations
Fast0Tag supports conventional tagging and annotation with previously unseen open-vocabulary tags, while running in constant time with respect to training-set size.
Source: §1 · p. 2The evidence is limited to the experimental vocabulary of seen tags; whether the principal direction is shared by unseen tags remains conditional on the tested hypothesis.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
The well-known word analogy experiments show that the recent word vectors capture fine-grained linguistic regularities in words by linear vector offsets, but it is unclear how well the simple vector offsets can encode visual regularities over words. We study a particular image-word relevance relation in this paper. Our results show that the word vectors of relevant tags for a given image rank ahead of the irrelevant tags, along a principal direction in the word vector space. Inspired by this observation, we propose to solve image tagging by estimating the principal direction for an image. Particularly, we exploit linear mappings and nonlinear deep neural networks to approximate the principal direction from an input image. We arrive at a quite versatile tagging model. It runs fast given a test image, in constant time w.r.t.\ the training set size. It not only gives superior performance for the conventional tagging task on the NUS-WIDE dataset, but also outperforms competitive baselines on annotating images with previously unseen tags
1. Introduction
The paper studies image tagging as a visual association between relevant and irrelevant words, finding that their word vectors can be ranked along a shared principal direction. It proposes Fast0Tag, which estimates that direction from an image and is designed to tag both seen and open-vocabulary unseen words.
1. Introduction
- Image tagging partitions a vocabulary into relevant and irrelevant tags for an image, a visual association distinct from the fine-grained relations tested by word analogies.The paper asks whether this coarser, descriptive visual association can be represented by linear offsets between relevant- and irrelevant-tag vectors.Source: §1 · p. 1
- Across image-induced visual association rules, relevant-tag and irrelevant-tag word-vector offsets align along a principal direction that ranks relevant words ahead.The paper reports that a direction w has positive inner products with the offsets between relevant and irrelevant tag vectors.Source: §1 · pp. 1–2
- Fast0Tag uses linear mappings or deep neural networks to estimate an image’s principal direction, enabling test-time tagging in constant time with respect to training-set size.The estimated direction is used to rank tag vectors, including open-vocabulary tags unseen during training, provided they share the word-vector space of the training tags.Source: §1 · p. 2
- Unlike prior methods that generally annotate only training-seen tags, the cited exception [19] requires unseen tags and test images to be known in advance and scales with 2^U combinations.The paper describes [19] as limited to a small number of unseen tags; Fast0Tag instead targets open-vocabulary unseen tags without prior knowledge of them.Source: §1 · p. 2
- Fast0Tag gives superior results over competitive baselines in conventional, zero-shot, and seen/unseen tagging scenarios.The paper distinguishes conventional tagging with seen tags, zero-shot tagging with many unseen tags, and tagging with both seen and unseen tags.Source: §1 · p. 2
2. Related work
Fast0Tag is positioned as a low-complexity image-tag ranking method that handles seen and unseen tags. The related work connects it to ranking-based tagging, word embeddings, and zero-shot multi-label learning.
Image tagging
- Fast0Tag shares FastTag’s low complexity, while using a ranking loss to align model learning with tag ranking.Earlier generative and nearest-neighbor methods ranked candidate tags, but nearest-neighbor approaches have high training and testing costs; FastTag was much faster and comparable to them.Source: §2 · p. 2
- Unlike prior methods limited to a closed vocabulary of seen tags, Fast0Tag ranks seen and arbitrary numbers of unseen tags; Fu et al. handle only a small fixed number of unseen tags.Fu et al. model all 2^U combinations of a pre-fixed set of U unseen tags.Source: §2 · pp. 2–3
Zero-shot learning
- Zero-shot classification learns from seen classes to classify unseen classes, using attributes and word vectors as semantic sources.It is distinguished from weakly supervised learning, which mines noisy samples for new concepts.Source: §2 · p. 3
- Fast0Tag directly models labels to rank many unseen tags, unlike Fu et al. [19], who treat each combination of multiple labels as a class.The work places Fast0Tag in zero-shot multi-label classification.Source: §2 · p. 3
3. The linear rank-ability of word vectors
The section finds that image-specific relevance among word vectors is expressed as a principal direction that ranks relevant tags ahead of irrelevant ones. Ranking SVM experiments show this pattern for seen tags and evidence that it transfers to unseen tags.
3.1. The regulation over words due to image tagging
- The seen-tag vocabulary is partitioned for each image into relevant and irrelevant tags, and unseen tags may be fixed or open and dynamically growing.The paper considers conventional tagging with seen tags, zero-shot tagging with unseen tags, and seen/unseen tagging that predicts from both sets.Source: §3.1 · p. 3
3.2. Principal direction and cluster structure
- Offsets between relevant and irrelevant tag vectors mostly point in a shared principal direction, while offsets to each relevant tag form clusters.The authors leave further investigation of the cluster structure for future research.Source: §3.2 · p. 3
3.3. Testing the linear rank-ability hypothesis
- In 300D GloVe and 300D, 500D, and 1000D word2vec spaces, more than two ranking SVMs with small λ yielded MiAP ≈1, indicating seen tags S were linearly rank-able under almost every visual association rule.Source: §3.3 · p. 4
- The learned directions ranked unseen tags significantly better than random ranking, supporting transfer of the seen-tag ranking direction to the unseen vocabulary.The test applied SVMs learned from seen-tag constraints to the 81 unseen tags; the paper notes that the experiment covers 1,006 words and calls for larger-scale and theoretical study.Source: Figure 3 · p. 4, §3.3 · pp. 4–5
4. Approximating the linear ranking functions
Fast0Tag predicts an image-specific principal direction in word-vector space and ranks tag vectors along it. It learns this direction from images using linear regression or a neural network trained directly on tag-ranking constraints.
4.1. Image tagging by ranking
- For a test image, Fast0Tag ranks tag word vectors along the predicted principal direction, covering both seen and unseen tags.The mapping from the image’s visual representation to word-vector space approximates the direction used for ranking.Source: §4.1 · p. 5
4.2. Approximation by linear regression
- Linear regression learns the mapping in two stages: ranking SVMs estimate directions from seen tags, then regression fits the mapping matrix to those estimates.The true principal directions are unknown, so the ranking-SVM directions serve as targets.Source: §4.2 · p. 5
- Unlike prior linear mappings for zero-shot classification and image annotation/classification, Fast0Tag’s mapped image vector represents a principal direction for its tags.The paper identifies this interpretation as a prominent difference from prior work.Source: §4.2 · p. 5
4.3. Approximation by neural networks
- The neural network uses two ReLU layers and a linear output layer to predict the principal direction, and is trained directly on ranking constraints rather than regressing to ranking-SVM directions.Regressing to those directions risks overfitting because training instances are few relative to network parameters, and the directions are not the true principal directions. The ranking loss is normalized per image by its number of constraints; practical regularization uses early stopping and 30% dropout.Source: §4.3 · pp. 5–6, Figure 4 · p. 5
- RankNet outperforms the Crammer-Singer and pairwise max-out losses by about 2% in MiAP, while hinge loss performs worst.The three pairwise ranking losses give comparable results; RankNet’s advantage may reflect easier control of optimization.Source: §4.3 · p. 6
5. Experiments on NUS-WIDE
On NUS-WIDE, Fast0Tag’s neural-network mapping outperforms the compared methods for conventional tagging and both Fast0Tag mappings perform best in zero-shot and seen/unseen tagging. The experiments also evaluate tagging with thousands of unseen tags and describe the dataset, features, and metrics used.
5.1. Dataset and configuration
- The evaluation uses 223,821 available NUS-WIDE images, split into 134,281 training and 89,603 test images, with 300D GloVe word vectors and ℓ2-normalized VGG-19 image features.The 81 high-quality groundtruth tags benchmark conventional tagging; metrics are MiAP and precision, recall, and F-1 for the top 3 and 5 tags.Source: §5.1 · p. 6
5.2. Conventional image tagging
- Fast0Tag’s neural-network mapping outperforms the compared methods for conventional tagging, while its linear mapping performs comparably to TagProp.Both Fast0Tag implementations have computation complexities as low as WARP and FastTag; TagProp has O(M^2) training and O(M) test complexity in training-set size M.Source: Table 1 · p. 7, §5.2 · p. 7
5.3. Zero-shot and Seen/Unseen image tagging
- Seen2Unseen, which extends a conventional tagging method using a ranking SVM in word-vector space, performs significantly better than random tag ranking.The method ranks seen tags first, learns from their ranking, and then ranks unseen tags for zero-shot tagging or seen and unseen tags for seen/unseen tagging.Source: §5.3 · p. 7
- Fast0Tag with either linear or neural-network mapping performs best on zero-shot and seen/unseen tagging.The evaluation uses 81 unseen tags and 925 seen tags.Source: §5.3 · p. 7, Table 2 · p. 8
- With 4,093 unseen tags, Fast0Tag’s results are reasonably low but significantly higher than random lists.The evaluation uses noisy Flickr annotations, which the authors say weaken its credibility.Source: Table 3 · p. 8, §5.3 · p. 8
- Fast0Tag’s zero-shot neural-network results are comparable to conventional-tagging results, and ranking-SVM results do not upper-bound the linear Fast0Tag model.The authors interpret this as evidence that learned ranking SVMs are not the “true” principal directions and that the neural network is an effective alternative for seeking them.Source: §5.3 · p. 8
6. Experiments on IAPRTC-12
On IAPRTC-12, Fast0Tag outperforms competitive baselines across conventional, zero-shot, and seen/unseen image tagging; the results also show a significant LabelEM+–LabelEM gap.
6.2. Results
- Fast0Tag outperforms competitive baselines across all three IAPRTC-12 tagging scenarios.Table 4 covers conventional tagging with 291 tags; Table 5 is also cited for the full set of scenarios.Source: Table 4 · p. 9, §6.2 · p. 9
6.1. Configuration
- The experiments use all 291 tags for conventional tagging and split the other tasks into 233 seen and 58 unseen tags.The dataset has 17,341 training images and 2,286 test images, with 15% of training images reserved for validation; the split follows.Source: §6 · p. 8, §6.1 · p. 9
6.2. Results
- A significant LabelEM+–LabelEM gap suggests traditional zero-shot classification methods are unsuitable for zero-shot and seen/unseen image tagging.Tweaking LabelEM and removing terms that compare identical images can improve performance.Source: §6.2 · p. 9
7. More qualitative results
The section presents qualitative tagging results on NUS-WIDE and IAPRTC-12, while noting that incomplete, noisy ground truth can make correct predictions appear mistaken.
7. More qualitative results
- Figure 6 shows qualitative tagging results on NUS-WIDE and IAPRTC-12.The NUS-WIDE examples supplement Figure 5 in the main text.Source: §7 · p. 9
- Incomplete and noisy tag ground truth can misclassify correct predictions, especially in 4k zero-shot tagging with many diverse candidates.The passage continues across two excerpts.Source: §7 · p. 9
8. Conclusion
The paper finds a principal direction in word-vector space along which relevant tags rank ahead of irrelevant tags for an image, and builds Fast0Tag to estimate that direction from images. The model is as efficient as FastTag and can annotate many previously unseen tags.
8. Conclusion
- For any image, relevant-tag vectors rank ahead of irrelevant-tag vectors along a principal direction in word-vector space.The experimental analyses involve 1,006 words; larger-scale and theoretical analyses are left for future work.Source: §8 · p. 9
- Fast0Tag estimates principal directions from input images to solve image tagging.The model is as efficient as FastTag, can annotate images with many previously unseen tags, and its effectiveness is validated by extensive experiments.Source: §8 · p. 9
- The study examines how visual relevance partitions words and whether vector offsets in word-vector space capture this association.The empirical analyses involve 1,006 words; larger-scale and theoretical analyses are required for future work.Source: §8 · p. 9
Cyclist Cycling Bike Jersey Road
Figure 6 presents top-five tag outputs for exemplar images from Fast0Tag across conventional, zero-shot, seen/unseen, and 4,093 zero-shot tagging tasks, alongside TagProp outputs for conventional tagging.
Cyclist Cycling Bike Jersey Road
- Figure 6 marks correct tags in green and mistaken tags in red italics.The exemplars are from [9](a) and [21](b).Source: Figure 6 · p. 10