Source-linked AI summary
Transitive Invariance for Self-supervised Visual Representation Learning
Xiaolong Wang, Kaiming He, Abhinav Gupta
TL;DR
The paper addresses the limited and complementary invariances learned by individual self-supervised tasks. It constructs a graph linking inter-instance and intra-instance variations, applies transitivity, and trains a Triplet-Siamese network. The resulting representations approach ImageNet-pretrained performance on object detection and outperform it on surface normal estimation, within the reported tasks.
Problem
Individual self-supervised tasks learn limited types of invariance, while naïvely combining multiple tasks provides little improvement.
Method
The method builds a graph with inter-instance and intra-instance edges, applies transitive reasoning, and trains a Triplet-Siamese network on the resulting invariant pairs.
Results
The learned representations approach ImageNet pre-training on object detection and outperform it on surface normal estimation, including 23.5% AP versus 24.4% AP on COCO.
Takeaways & Limitations
Combining complementary invariances through data organization and transitivity yields representations competitive with ImageNet pre-training across reported recognition tasks and stronger for surface normal estimation.
Takeaways & Limitations
Context prediction can fail under drastic pose, viewpoint, or deformation changes unless training data contains sufficiently gradual and diverse variations.
Abstract
from arXiv · showhide
Learning visual representations with self-supervised learning has become popular in computer vision. The idea is to design auxiliary tasks where labels are free to obtain. Most of these tasks end up providing data to learn specific kinds of invariance useful for recognition. In this paper, we propose to exploit different self-supervised approaches to learn representations invariant to (i) inter-instance variations (two objects in the same class should have similar features) and (ii) intra-instance variations (viewpoint, pose, deformations, illumination, etc). Instead of combining two approaches with multi-task learning, we argue to organize and reason the data with multiple variations. Specifically, we propose to generate a graph with millions of objects mined from hundreds of thousands of videos. The objects are connected by two types of edges which correspond to two types of invariance: "different instances but a similar viewpoint and category" and "different viewpoints of the same instance". By applying simple transitivity on the graph with these edges, we can obtain pairs of images exhibiting richer visual invariance. We use this data to train a Triplet-Siamese network with VGG16 as the base architecture and apply the learned representations to different recognition tasks. For object detection, we achieve 63.2% mAP on PASCAL VOC 2007 using Fast R-CNN (compare to 67.3% with ImageNet pre-training). For the challenging COCO dataset, our method is surprisingly close (23.5%) to the ImageNet-supervised counterpart (24.4%) using the Faster R-CNN framework. We also show that our network can perform significantly better than the ImageNet network in the surface normal estimation task.
1. Introduction
Self-supervised tasks provide free supervision but often learn limited types of visual invariance, and naïvely combining tasks yields little improvement. The paper instead uses transitive relations over data to learn richer invariance, achieving competitive transfer results without annotations.
- Motivation: Self-supervised learning mines invariance from data without expensive human annotations.These methods differ from supervised learning, which learns invariance from semantic labels.
- Motivation: Naïvely combining multiple self-supervised tasks provides little improvement over individual approaches.
- Approach: The method builds a graph with inter-instance and intra-instance edges, then applies transitivity to infer richer invariance.The resulting data trains a Triplet-Siamese network.
- Results: 63.2% mAP is achieved on PASCAL VOC 2007 with Fast R-CNN, compared with 67.3% for ImageNet pre-training.
- Results: 23.5% AP is achieved on COCO with Faster R-CNN, compared with 24.4% AP for the ImageNet-pretrained counterpart.
2. Related Work
Prior work learns visual invariance through generative models or self-supervised tasks, including temporal tracking and context prediction. The paper combines complementary inter-instance and intra-instance signals because each individual approach has a distinct limitation.
- Prior approaches: Unsupervised representation learning includes generative models and self-supervised methods for learning invariant features.Generative approaches include auto-encoders, RBMs, GANs, and VAEs; self-supervised approaches exploit structure in images or videos.
- Combined framework: The framework uses context prediction for inter-instance invariance and tracking for intra-instance invariance.
- Graph construction: The graph construction clusters object nodes and links samples within child clusters by inter-instance edges, while tracked views receive intra-instance edges.
- Inter-instance invariance: Context prediction learns spatial commonality across instances but can fail under drastic pose, viewpoint, or deformation changes.
- Intra-instance invariance: Visual tracking captures viewpoint and deformation changes within an identical instance but cannot associate different instances sharing the same semantics.
3. Overview
The paper represents image-patch affinities as a graph with two edge types and uses transitivity to create new invariant pairs. A Triplet-Siamese network then learns similar representations for related samples while separating unrelated ones.
- Goal: The goal is to capture both inter-instance invariance and intra-instance invariance, which multi-task learning did not learn satisfactorily.
- Graph representation: The graph contains image-patch nodes connected by inter-instance edges for different similar-looking objects and intra-instance edges for the same object across time.
- Transitive reasoning: Transitivity links previously unconnected pairs such as ⟨A′, B′⟩, ⟨A, B′⟩, and ⟨A′, B⟩.
- Representation learning: The Triplet-Siamese network brings invariant samples together and separates them from an unrelated distractor, using VGG16 as its backbone.
4. Graph Construction
The graph combines millions of video-mined object patches with inter-instance edges from feature-based clustering and intra-instance edges from visual tracking. Hierarchical clustering creates small child clusters whose members are linked as inter-instance neighbors.
- Graph nodes: Millions of moving objects are mined from 100K unlabeled videos and used as graph nodes.The objects are extracted using motion cues, and their image patches form the graph nodes.
- Inter-instance edges: Inter-instance edges connect nearby image patches in a feature space and are organized into small, high-purity object clusters.The features come from a context-prediction representation, and nearby nodes are linked to represent commonalities across instances.
- Inter-instance edges: The clustering pipeline first groups patches hierarchically, then searches within parent clusters for mutually near neighbors.This reduces the cost of finding many small clusters among millions of patches.
- Inter-instance edges: Kmeans with K = 5000 is followed by nearest-neighbor grouping into child clusters of four mutually top-10-nearest samples.Clusters smaller than 100 are removed after the first stage; child clusters may overlap.
- Intra-instance edges: Intra-instance edges link an object patch to another sample obtained by tracking the same moving object for N = 30 frames.The tracker is unsupervised and supplies viewpoint and deformation changes of the same instance.
5. Learning with Transitions in the Graph
The method applies transitivity to inter-instance and intra-instance graph edges to create richer positive pairs, then trains a Triplet-Siamese VGG16 network with ranking loss. The learned representations can encode both same-instance viewpoint invariance and semantic similarity across different objects.
- Graph transitions: Transitivity links A, B, A′, and B′ to create additional pairs combining inter-instance and intra-instance invariance.If A-B is an inter-instance edge and A-A′ plus B-B′ are intra-instance edges, pairs such as A-B′ and A′-B become related.
- Positive pairs: The network is trained to produce similar representations for directly linked or transitively linked patches.These related samples are treated as positive pairs for representation learning.
- Triplet-Siamese training: A Triplet-Siamese VGG16 network uses shared-weight towers and 1024-d features for triplets containing positive and negative pairs.The first two patches form a positive pair, while the negative pair uses an unrelated patch.
- Triplet-Siamese training: The ranking loss makes related samples closer than unrelated samples, using cosine-based feature distance and a margin of 0.5.Different combinations of A, B, A′, and B′ train viewpoint invariance and semantic invariance across objects.
- Additional relation: Enforcing an additional ordering between same-instance and cross-instance distances produced no improvement, although training generally satisfied that ordering.The tested relation was D(A, A′) < D(A, B′).
6. Experiments
Experiments evaluate the learned representations as fixed feature extractors and after transfer to object detection and surface normal estimation. The method captures varied object appearances and transfers competitively across detection benchmarks, while outperforming ImageNet pre-training for surface normal estimation.
- Qualitative Results without Fine-tuning: The learned features represent greater viewpoint variety within object categories and activate on semantic parts such as vehicle fronts, wheels, animal legs, eyes, and faces.Nearest-neighbor retrieval uses VGG16 pool5 features without fine-tuning; convolutional-unit visualizations show responses associated with different semantic object parts.
- Analysis on Object Detection: The combined method outperformed the individual self-supervised approaches, indicating that exploiting a greater variety of invariance improves representation learning.The experiments compare the proposed method with context prediction and tracking-based alternatives.
- Object Detection with Faster R-CNN: 23.5% AP on COCO was within 1% of the 24.4% AP obtained by fine-tuning Faster R-CNN from ImageNet pre-training.Using the shorter ImageNet schedule produced 23.7% AP, comparable to the self-supervised result.
- Adapting to Surface Normal Estimation: Surface normal estimation was 3 to 4% better than ImageNet pre-training on most reported metrics, whereas the related self-supervised networks were slightly worse than ImageNet.The results include mean and median angular error and the percentage of pixels below 11.25, 22.5, and 30 degrees.