Source-linked AI summary
Unsupervised Learning of Visual Representations using Videos
Xiaolong Wang, Abhinav Gupta
TL;DR
The paper asks whether CNN representations require massive semantic supervision. It learns them from unlabeled videos by treating visual tracking as supervision, achieving 52% mAP on VOC 2012 versus 54.4% for an ImageNet-supervised ensemble.
Problem
The paper asks whether CNNs need millions of semantically labeled images and strong supervision to learn good visual representations.
Method
The method tracks patches in unlabeled videos and trains a Siamese-triplet CNN with ranking loss to bring tracked patches closer than random patches.
Results
52% mAP is achieved on VOC 2012 without ImageNet images or bounding box regression, compared with 54.4% mAP for an ImageNet-supervised ensemble.
Takeaways & Limitations
Unlabeled videos can support CNN representation learning that approaches ImageNet-supervised performance on object detection and performs competitively on other tasks.
Abstract
from arXiv · showhide
Is strong supervision necessary for learning a good visual representation? Do we really need millions of semantically-labeled images to train a Convolutional Neural Network (CNN)? In this paper, we present a simple yet surprisingly powerful approach for unsupervised learning of CNN. Specifically, we use hundreds of thousands of unlabeled videos from the web to learn visual representations. Our key idea is that visual tracking provides the supervision. That is, two patches connected by a track should have similar visual representation in deep feature space since they probably belong to the same object or object part. We design a Siamese-triplet network with a ranking loss function to train this CNN representation. Without using a single image from ImageNet, just using 100K unlabeled videos and the VOC 2012 dataset, we train an ensemble of unsupervised networks that achieves 52% mAP (no bounding box regression). This performance comes tantalizingly close to its ImageNet-supervised counterpart, an ensemble which achieves a mAP of 54.4%. We also show that our unsupervised network can perform competitively in other tasks such as surface-normal estimation.
1. Introduction
The introduction questions whether CNNs require millions of semantically labeled images and proposes learning representations from unlabeled web videos using visual tracking.
- CNNs typically learn visual representations through strongly supervised training on millions of labeled and augmented examples.
- Earlier unsupervised approaches using static images or video frames often learned V1-like filters but remained far behind supervised methods on object detection.
- Videos may provide richer learning signals than static images because dynamic sensory inputs expose temporal relationships between views of visual content.
- The paper tracks millions of patches in hundreds of thousands of unlabeled videos and trains a Siamese-triplet network with ranking loss.
- 52% mAP is achieved on VOC 2012 without ImageNet images or bounding box regression, approaching the 54.4% mAP of an ImageNet-supervised ensemble.
2. Related Work
Prior unsupervised representation learning spans generative, hand-crafted-feature, deep pixel-based, and video-based approaches, while this work connects video tracking with deep metric learning.
- Unsupervised representation methods include generative models with priors, clustering over manually defined features, and deep learning directly from pixels.
- Early deep unsupervised methods sought sparse, reconstructive representations and demonstrated V1-like filters, initially with single-layer models.
- Video-based approaches imposed temporal smoothness, slowness priors, paired-frame modeling, or future-frame prediction to learn representations.
- Deep metric-learning methods used Siamese or triplet architectures for similarity, verification, hashing, or retrieval, but required labeled data.
3. Overview
The overview replaces semantic labels with temporal correspondence: tracked patches from the same moving instance should be close in feature space, while a third patch prevents collapse.
- The method uses temporal information from videos as supervisory information because nearby frames commonly preserve scenes and object instances.
- Millions of patches are tracked over time, making the first and last patches of a track positive examples that should remain close in feature space.
- A randomly sampled third patch forms a triplet and prevents the degenerate solution in which all patches map to one feature-space point.
- Hard-negative selection chooses the patch violating the constraint most strongly, producing more meaningful gradients after easy triplets converge.
4. Patch Mining in Videos
Patch mining identifies moving regions in noisy videos, selects a bounding box around moving SURF points, and tracks the resulting patch to generate CNN training pairs.
- Optical-flow magnitude alone is unreliable in YouTube videos because camera motion makes moving objects difficult to localize.
- The pipeline first extracts SURF interest points and uses trajectories to distinguish motion patterns before selecting candidate regions.
- Tracking the selected region provides the first and last frames of a track as paired patches for CNN training.
- The selected 227 × 227 bounding box contains most moving SURF points within 448 × 600 frames and may contain an object or part of one.
5. Learning Via Videos
The method learns CNN representations from tracked video patches by making tracked pairs similar and random negatives dissimilar in a shared feature space. A Siamese-triplet network, ranking loss, and hard-negative mining turn these relationships into trainable supervision.
- Patch Mining: Millions of tracked patch pairs from videos provide training instances without semantic labels.The method mines patch pairs by tracking patches over time, treating the first and last frames as the same object instance or object part.
- Siamese-Triplet Network: The Siamese-triplet network uses three parameter-sharing base networks to compare a query, its tracked match, and a negative patch.The base network follows AlexNet’s convolutional layers and produces a 1024-dimensional feature after two fully connected layers.
- Ranking Loss Function: Cosine feature distance and hinge ranking loss encourage the query to be closer to its tracked patch than to a random negative by a margin.The experiment sets the margin M = 0.5.
- Hard Negative Mining: Hard-negative mining selects the negative patches with the highest losses, producing more meaningful gradients after initial random-negative training.After 10 epochs of random selection, the method chooses the top K negatives by loss, using K = 4 in the experiments.
- Transfer Learning: The learned CNN is transferred to object detection and surface-normal estimation by adapting the base network and supervised outputs.One transfer strategy initializes convolutional layers from the unsupervised network, randomly initializes fully connected layers, and uses an increased learning rate for fine-tuning.
- Transfer Learning: Iterative fine-tuning reuses the unsupervised ranking task after supervised adaptation to recover additional value from the millions of unsupervised training examples.The procedure first fine-tunes on PASCAL VOC data, then re-adapts the resulting network to the ranking triplet task.
6. Experiments
Experiments evaluate the unsupervised representation without fine-tuning and after transfer to object detection and surface-normal estimation. The representation supports nearest-neighbor retrieval, improves VOC detection over scratch training, benefits from more data and ensembling, and approaches ImageNet performance on both transferred tasks.
- 6.1. Unsupervised CNNs without Fine-tuning: The unsupervised CNN achieves 40% top-20 retrieval rate on VOC 2012, exceeding HOG-based ELDA at 24% and random AlexNet at 19%.ImageNet CNN reaches 62%, but it uses semantic supervision.
- 6.1. Unsupervised CNNs without Fine-tuning: 41% MIT Indoor 67 classification accuracy from unsupervised pool5 features exceeds GIST+SVM at 21% and random AlexNet at 16%.ImageNet-trained AlexNet reaches 54% accuracy.
- 6.2.1 Object Detection: 47.5% VOC 2012 mAP follows fine-tuning an unsupervised network pretrained on 8M patches, compared with 44% from training from scratch.Increasing pretraining data from 1.5M to 5M and 8M patches raises single-model mAP from 46.2% to 47% and 47.5%.
- 6.2.1 Object Detection: 52% VOC 2012 mAP is obtained by ensembling three fine-tuned unsupervised networks pretrained on 1.5M, 5M, and 8M patches.The corresponding three-network ImageNet ensemble reaches 54.4% mAP.
- 6.2.1 Object Detection: 48% mAP results from iterative fine-tuning, which re-adapts the VOC-fine-tuned network to the unsupervised triplet task before fine-tuning again.This is 1% better than the initial fine-tuned network.
- 6.2.2 Surface Normal Estimation: Surface-normal estimation after unsupervised pretraining and fine-tuning significantly outperforms training from scratch and comes within approximately 1% of ImageNet-pretrained CNNs.Evaluation uses mean and median angular error plus pixel percentages below 11.25, 22.5, and 30 degrees on NYUv2.
7. Discussion and Conclusion
The video-based unsupervised CNN approach outperforms training from scratch and approaches ImageNet-pretrained performance, suggesting semantic supervision may not be essential for effective CNN training.
- 3.5% higher performance than CNNs trained from scratch was achieved after fine-tuning the unsupervised pre-trained CNN on VOC training data.
- 4.7% improvement over scratch was achieved by the ensemble version of the unsupervised approach.
- Within 2.5% of an ImageNet-pretrained CNN, the ensemble approached supervised pre-training performance.