Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions

Jimmy Ba; Kevin Swersky; Sanja Fidler; Ruslan Salakhutdinov

Source-linked AI summary

Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions

Jimmy Ba, Kevin Swersky, Sanja Fidler, Ruslan Salakhutdinov

arXiv:1506.00511v2Jun 1, 2015cs.LGcs.CVcs.NE

TL;DR

Zero-shot visual recognition needs semantic information for unseen categories without costly, manually defined attributes. The paper predicts CNN classifier weights from encyclopedia text at fully connected and convolutional layers, and reports stronger results than previous methods while discovering pseudo-attributes.

Problem
Collecting semantic attributes and fine-grained annotations for visual categories is difficult, costly, and time-consuming.
Method
The model uses encyclopedia articles and image pixels to predict classifier weights for fully connected and convolutional CNN layers.
Results
The model significantly outperforms previous zero-shot methods on Caltech-UCSD bird and Oxford flower datasets and discovers class-specific pseudo-attributes.
Takeaways & Limitations
Natural-language descriptions can provide useful semantic information for predicting unseen visual classes and generating pseudo-attributes.
Takeaways & Limitations
Zero-shot multi-class recognition remains around 10% top-1 accuracy on CUBird and lower than some attribute-based methods.

Abstract

from arXiv · show

One of the main challenges in Zero-Shot Learning of visual categories is gathering semantic attributes to accompany images. Recent work has shown that learning from textual descriptions, such as Wikipedia articles, avoids the problem of having to explicitly define these attributes. We present a new model that can classify unseen categories from their textual description. Specifically, we use text features to predict the output weights of both the convolutional and the fully connected layers in a deep convolutional neural network (CNN). We take advantage of the architecture of CNNs and learn features at different layers, rather than just learning an embedding space for both modalities, as is common with existing approaches. The proposed model also allows us to automatically generate a list of pseudo- attributes for each visual category consisting of words from Wikipedia articles. We train our models end-to-end us- ing the Caltech-UCSD bird and flower datasets and evaluate both ROC and Precision-Recall curves. Our empirical results show that the proposed model significantly outperforms previous methods.

1. Introduction

The paper addresses the cost and limited scalability of collecting visual annotations by using encyclopedia text to predict classifiers for unseen visual categories. Its deep architecture predicts weights at multiple CNN levels and achieves stronger zero-shot results while discovering pseudo-attributes.

Large image datasets and fine-grained annotations are expensive and time-consuming to collect.
Wikipedia and other online encyclopedias provide a large source of textual knowledge for training visual models with minimal additional annotation.
The model uses encyclopedia articles to generate classifiers for unseen categories, avoiding manually defined attributes and using raw text and image pixels.
Text predicts classifier weights for both fully connected and convolutional CNN layers, with convolutional filters operating on intermediate feature maps.
The model evaluates ROC-AUC, mean Average-Precision, and classification accuracy, with different objective functions outperforming one another under different metrics.
On Caltech-UCSD birds and Oxford flowers, the proposed model significantly outperforms the previous state-of-the-art and can automatically discover class-specific pseudo-attributes.

2. Related work

Related zero-shot methods share semantic information between images and labels or use manually annotated attributes. This paper instead derives features from natural-language descriptions and uses deep networks to jointly embed text and images.

Domain adaptation transfers information between source and target datasets, whereas zero-shot learning involves previously unseen class labels.
Semantic label embedding jointly trains image and label projections so semantically related labels and images are nearby in a shared space.
Prior methods use ranking objectives, skip-gram label features, or CNN classifier uncertainty to exploit semantic relationships between labels.
The paper differs by modeling relationships between attributes and images rather than relationships between labels.
Earlier attribute-based approaches rely on manually annotated attributes, while this work derives embedding features directly from natural-language descriptions.
Its architecture jointly embeds image and text features with deep neural networks rather than using probabilistic regression with domain adaptation.

3. Predicting a classifier

The model learns classifiers for unseen visual classes by predicting classifier weights from class text descriptions. It extends this idea across fully connected and convolutional CNN layers, using dimensionality reduction and spatially pooled convolutional scores.

Predicting a classifier: Text features predict classifier weights for previously unseen classes, replacing static class-specific weights with a learned function of each class description.
Predicting a classifier: A visual mapping projects image features into a lower-dimensional space, reducing the dimensionality of the predicted weight vector.The reduced formulation uses neural networks for both text-to-weight and image-to-feature mappings.
Predicting a classifier: The convolutional classifier predicts filters from text, applies them to reduced CNN feature maps, and uses global pooling to produce a class score.Global pooling aggregates local responses across the image; different pooling choices change sensitivity to local features.
Predicting a classifier: The joint model combines intermediate convolutional features with final fully connected features to exploit both spatially local and high-level representations.Convolutional layers retain spatial structure and use fewer, locally shared weights than fully connected layers.

4. Learning

The paper trains text-conditioned classifiers with binary cross entropy or hinge loss objectives. Minibatch approximation addresses the cost of evaluating every image against every class.

Binary Cross Entropy: Binary cross entropy uses a 0-1 image-class indicator and evaluates predicted binary-classifier scores for image-text pairs.The indicator identifies whether an image belongs to the class represented by the text.
Binary Cross Entropy: Minibatch training reduces computational cost from O(N × C) to O(B × B) by summing over sampled images and their labels.
Hinge Loss: Hinge loss uses a {1, −1} encoding and encourages higher scores for the correct description than for other classes.Its margin ϵ is typically set to 1, and minibatch training can also be adapted to this objective.
Euclidean Distance: The Euclidean-distance objective can be obtained from the classifier formulation, while hinge loss avoids infinite penalties on negative examples during joint embedding learning.

5. Experiments

Experiments compare fully connected, convolutional, and joint models using ROC-AUC, PR-AUC, and top-K accuracy. The models use fixed ImageNet-pretrained VGG features and neural text-to-weight mappings.

Models and evaluation: The fc model predicts fully connected weights, the conv model predicts convolutional filters, and fc+conv jointly combines both classifiers.
Models and evaluation: ROC-AUC, PR-AUC(AP), and top-K classification accuracy evaluate binary ranking, imbalanced-data retrieval quality, and multiclass predictions.
Implementation: Image features come from a 19-layer VGG network pretrained on ImageNet without fine-tuning, using fc1 activations and conv5_3 feature maps.
Implementation: Text and visual transformations use ReLU neural networks, while convolutional features are reduced before applying text-predicted filters.The reported architecture includes 300-unit hidden layers and K′ = 5 reduced feature maps.
Implementation: Adam trains the models with minibatches of 200 images because SGD performs poorly, potentially due to sparse text-feature gradients and dense convolutional gradients.The fully connected model trains in 1–2 hours, while the joint model takes 4 hours on a GTX Titan.

5.2. Caltech UCSD Bird

The bird experiments use raw images and Wikipedia articles rather than semantic parts or attributes. Evaluation covers two CUB versions under a zero-shot split, with ROC-AUC and PR-AUC reported against other methods.

Dataset and text representation: The method is evaluated on CUB200-2010 and CUB200-2011 using raw bird images and Wikipedia articles instead of semantic parts and attributes.
Dataset and text representation: Each of the 200 bird classes has one Wikipedia article averaging about 400 words, represented as a 9763-dimensional tf-idf vector.Log normalization helps because article lengths vary substantially across classes.
Results: Table 1 reports ROC-AUC and PR-AUC(AP) for zero-shot unseen classes and seen-class test data, including class-averaged mean AUCs.For each metric, the table reports the best result among models trained with different objective functions.
Experimental split: CUB200-2010 contains 6033 images, with 40 classes held out as unseen and the remaining 160 classes divided into training and seen-class testing images.The evaluation uses 5-fold cross-validation and follows the protocol of the prior work used for comparison.

5.3. Oxford Flower

On Oxford Flower-102, the model uses 82 classes for training and 20 unseen classes for testing, achieving a substantial improvement over prior zero-shot results.

Oxford Flower-102 contains 102 classes and 8,189 images, with roughly 40 to 260 images per class.
The experimental split uses 82 flower classes for training and 20 classes as unseen categories during testing.
The proposed method dramatically improves upon the state of the art for zero-shot learning on the Oxford Flower dataset.

5.5. Effect of objective functions

The study compares objective functions using several evaluation metrics and examines how intermediate CNN layers affect the joint model. BCE provides the best overall metric balance, while deeper convolutional features are more discriminative.

Objective functions: Binary cross entropy provides a good balance among ROC-AUC, PR-AUC, and classification accuracy.
Objective functions: Hinge-loss models consistently achieve the strongest PR-AUC but perform poorly on top-K accuracy for zero-shot classes.
Objective functions: BCE shows the best overall performance across ROC-AUC, PR-AUC, and classification accuracy.
Convolutional features: The convolutional classifier is evaluated with intermediate VGG layers, and conv5_3 features are more discriminative than lower conv4_3 features.

5.7. Learning on the full datasets

The model can also be trained on complete datasets for conventional object recognition. Its text-driven classifier analyzes unseen classes and derives visually meaningful pseudo-attributes from descriptions.

The proposed method can perform object recognition when trained on an entire dataset, with fine-tuning results reported in Table 4.
At test time, the model classifies images from unseen visual classes using their textual descriptions alongside images from trained classes.
Word sensitivity is estimated by zeroing each word’s TF-IDF entry, renormalizing input magnitude, and measuring the resulting PR-AUC change.
Words causing the largest PR-AUC decreases are treated as approximate class-specific pseudo-attributes.
Sensitive words can identify bird types, physical descriptors, or rare nond descriptive terms, depending on the class.
Projecting class text into the visual feature space retrieves within-class and cross-class image neighbors with similar physical characteristics.

6. Limitations

Although the method substantially improves ROC-AUC, its zero-shot multiclass recognition accuracy remains below some attribute-based approaches.

Despite significant ROC-AUC improvement, zero-shot multiclass recognition remains lower than some attribute-based methods.On CUBird, top-1 accuracy is around 10%.
The authors suggest using discovered attribute lists in a second-stage attribute-classification model as future work.

7. Conclusion

The paper concludes that encyclopedia articles can support zero-shot prediction of unseen image classes by producing classifier weights from natural language. Its CNN-based approach uses multiple feature levels, improves reported zero-shot performance, and learns pseudo-attributes from articles, while future work proposes replacing TF-IDF with an LSTM.

The model predicts unseen image classes from encyclopedia articles by using natural language descriptions to produce classifier weights.
The approach incorporates intermediate convolutional feature maps together with the final fully connected feature vector.
The method significantly outperforms previous zero-shot methods on ROC-AUC and substantially improves state-of-the-art performance on CUBird and Oxford Flower using raw images and text articles.
The network learns class-specific pseudo-attributes from encyclopedia articles, and text embeddings capture useful semantic information in images.
Future work plans to replace TF-IDF text features with an LSTM recurrent neural network.

8. Appendix

The appendix visualizes how text-derived representations relate to images and predicted convolutional filters across bird and flower datasets. It includes word-sensitivity analyses, nearest-image queries, filter activations, deconvolution visualizations, and example encyclopedia descriptions.

Figure 3: CUB200-2010: Figure 3 analyzes word sensitivities for unseen CUB200-2010 classes using the fully connected model and shows within-class and overall nearest image neighbors.TF-IDF entries are independently zeroed, and the five words causing the largest PR-AUC reductions are displayed as approximate attributes.
Figure 4: Oxford Flower: Figure 4 applies the same word-sensitivity and nearest-neighbor visualization to unseen Oxford Flower classes.The left panels identify words whose removal most reduces test-set PR-AUC; the right panels show image neighbors for projected article vectors.
Figures 5–6: CUB200-2010 filter visualizations: Figures 5 and 6 visualize predicted convolutional filters for unseen bird articles using a model trained on CUB200-2010.They show the five validation images with highest activations, including seen and unseen classes, with filter visualizations generated by deconvolution.
Example encyclopedia descriptions: The appendix includes encyclopedia passages describing bird and flower classes used alongside example images and model visualizations.Examples include Nashville Warbler, Red-faced Cormorant, Scarlet Tanager, Rose-breasted Grosbeak, Erysimum, and Fritillaria.
Figure 7: Oxford Flower filter visualizations: Figure 7 visualizes predicted convolutional filters for unseen flower articles using a model trained on Oxford Flower.The figure shows the five highest-activation validation images, including seen and unseen classes, alongside deconvolution visualizations.