Source-linked AI summary
Self-supervised learning of visual features through embedding images into text topic spaces
Lluis Gomez, Yash Patel, Marçal Rusiñol, Dimosthenis Karatzas, C. V. Jawahar
TL;DR
Large-scale annotated datasets require substantial human effort, motivating alternatives that use the text accompanying images. This paper trains a CNN to predict text-derived topic semantics from images, and reports comparable state-of-the-art performance across image classification, object detection, and multi-modal retrieval.
Problem
The paper seeks an alternative to fully supervised CNN training because large-scale annotated datasets require substantial human resources, and asks whether illustrated content can provide supervision without human annotation.
Source: §1 · p. 1Method
The method mines correlated texts and images from illustrated articles, represents text as LDA topic probabilities, and trains a CNN to predict those probabilities from image pixels.
Source: §1 · p. 2, §3.1 · p. 3Results
The learned features are comparable with state-of-the-art self-supervised algorithms on image classification, object detection, and multi-modal retrieval tasks.
Source: §5 · p. 8Takeaways & Limitations
The approach learns visual features from scratch without a large annotated dataset and supports multi-modal retrieval without extra annotation or learning efforts.
Source: §1 · p. 2The framework assumes that associated textual information is generated by a mixture of hidden topics.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.
1. Introduction
The paper asks whether freely available illustrated articles can provide supervision for visual learning without extensive human annotation. It proposes predicting article-level topic semantics from images, with the aim of learning generic features that remain useful for specific vision tasks.
1. Introduction
- Topic-level supervision can draw on broader categories with hundreds or thousands of images, where specific entities may have only around five.The paper argues that resulting features are generic for a topic but useful for more specific vision tasks.Source: Figure 2 · p. 2, §1 · p. 2
- The method uses text from illustrated articles as noisy, freely available image annotation, avoiding reliance on manually collected labels.The paper frames this as an alternative to fully supervised CNN training, which depends on substantial annotated data.Source: §1 · pp. 1–2
- The CNN predicts an article’s semantic context from its image, using LDA topic probabilities as the textual supervisory signal.The paper proposes mining correlated text and images from Wikipedia articles.Source: §1 · p. 2
- The authors report comparable or better performance than recent self-supervised and unsupervised methods in image classification, object detection, and multi-modal retrieval.The contribution statement places these results on standard benchmarks.Source: §1 · p. 2
2. Related Work
Prior unsupervised approaches and non-visual supervisory signals provide context for the paper’s use of text. Its distinction is to use topic-level text semantics to train a CNN, rather than only embedding image and text representations in a shared retrieval space.
2. Related Work
- Earlier data-dependent unsupervised methods include sparse coding, RBMs, auto-encoders, and K-means, but the passage says no generic method works well with real-world images.Self-supervised approaches instead use signals such as egomotion, object motion, patch position, or sound.Source: §2 · p. 2
- The paper explores text as a CNN self-supervision modality, using topic models to capture semantics in abundant illustrated web articles without human supervision.The authors position this as a larger level of abstraction than conventional image annotation.Source: §2 · pp. 2–3
- Prior cross-modal retrieval methods use topic models or CCA to relate modalities, while this method uses LDA topic probabilities to supervise CNN feature learning.The learned generic visual features are intended for use in other vision tasks.Source: §2 · p. 3
3. TextTopicNet
TextTopicNet uses LDA to represent article text as topic probabilities, then trains a CNN to predict those distributions from image pixels. The resulting features are proposed for reuse in visual tasks and retrieval.
3. TextTopicNet
- TextTopicNet first learns text topics from paired articles and images, then trains a CNN to predict article topic-probability distributions from image pixels.Source: §3 · p. 3
3.1. LDA topic modeling
- LDA represents each article as a probability distribution over latent topics, providing a compact, semantically interpretable alternative to word-level representations.Source: §3.1 · p. 3
3.2. Training a CNN to predict semantic topics
- Training uses English ImageCLEF articles, filtered to 100,785 images and 35,582 articles; the CNN is trained from raw articles and images without human annotations.Source: §3.2 · p. 4
- The discovered topics correspond to broad categories, and their representative images show regularities despite noise from other images in the same topic.Source: §3.2 · pp. 4–5, Figure 4 · p. 4
- The CNN predicts overlapping topic probabilities rather than a single topic; sigmoid cross-entropy is chosen to match that multi-label target.Source: §3.2 · pp. 4–5
3.3. Self-supervised learning of visual features
- TextTopicNet features are proposed as convolutional feature extractors and pretraining for visual tasks, while the shared LDA-CNN topic space also supports retrieval and annotation.Source: §3.3 · p. 5
4. Experiments
The experiments assess TextTopicNet features across image classification, object detection, and multimodal retrieval. They compare topic-supervised features with other approaches and examine both feature behavior and retrieval results.
4. Experiments
- The experiments use one LDA topic model learned from 35,582 English Wikipedia articles, with topic count selected empirically using validation data.The processed corpus has a 50,913-word dictionary; 40 topics gives the best validation performance reported for fc7 features.Source: §4 · p. 5, §4.1 · p. 5, Figure 5 · p. 5
4.1. Unsupervised feature learning for image classification
- Text semantics as supervision outperform all other modalities in the PASCAL VOC2007 image-classification experiment.The comparison is reported for Table 2; pool5 features are also noted as more discriminative for difficult classes such as bottle, pottedplant, and cow.Source: §4.1 · p. 5, Table 2 · p. 6
- TextTopicNet (Wiki) performs surprisingly well compared with TextTopicNet (COCO), although more specific textual content helps the model learn better features.The COCO annotations are human-generated, making that entry weakly supervised rather than self-supervised.Source: §4.1 · p. 6
- Concatenating TextTopicNet (Wiki) and Sound fc7 features raises mAP to 54.81%, indicating some complementarity between the supervisory signals.The experiment combines the two models’ fc7 features.Source: §4.1 · p. 6
- TextTopicNet units are mostly selective to generic textures, shapes, and object-parts, with some object-selective units also present.The fc7 visualizations include units selective to textures and some specific objects, such as faces.Source: §4.1 · p. 6, Figure 6 · p. 6
4.2. Comparison to unsupervised pre-training and semi-supervised methods
- The fine-tuning experiments cover PASCAL image classification and object detection, plus STL-10 classification; the STL-10 experiment does not use its 100,000 unlabeled images.The supplied passages describe the comparisons and setup but do not provide the resulting performance values.Source: §4.2 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
4.3. Multi-modal image retrieval
- On Wikipedia retrieval, the self-supervised method outperforms unsupervised approaches and performs competitively with supervised methods without labeled data.Images and documents are projected into the learned topic space, and retrieval ranks entities by KL-divergence.Source: §4.3 · p. 7, Table 5 · p. 7
- For image queries, topic-space retrieval returns semantically close results that need not look visually similar; earlier-layer features tend to retrieve more visually similar images.The query images are from PASCAL VOC 2007 and were not seen during training.Source: §4.3 · pp. 7–8, Figure 7 · p. 8
5. Conclusion
The conclusion presents a self-supervised method that treats text in illustrated articles as noisy image annotations and trains a CNN to predict semantic context. Its generic features are reported as useful across several vision tasks and comparable with state-of-the-art self-supervised algorithms.
5. Conclusion
- The method learns visual features by training a CNN to predict the semantic context in which an image is more likely to appear as an illustration.Text in illustrated articles is treated as noisy image annotation.Source: §5 · p. 8
- Although the learned features are generic for broad topics, the experiments use them for image classification, object detection, and multimodal retrieval.Source: §5 · p. 8
- The results are comparable with state-of-the-art self-supervised algorithms for visual feature learning.Source: §5 · p. 8