Source-linked AI summary
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Senthil Purushwalkam, Abhinav Gupta
TL;DR
The paper asks why contrastive self-supervised representations perform well despite unclear encoded invariances and dependence on aggressive augmentation. It measures task-relevant invariances, analyzes augmentation and dataset bias, and uses temporal transformations in videos to improve representations. The video-based approach achieves stronger invariance measures and downstream object-recognition performance than MoCo-v2 trained on the same data.
Problem
The paper addresses limited understanding of why contrastive self-supervised representations work and which task-relevant invariances they encode.
Method
The paper measures task-relevant invariances, examines augmentation and object-centric dataset bias, and trains representations using naturally occurring temporal transformations in videos.
Results
Video-trained representations outperform MoCo-v2 trained on the same data in viewpoint, category instance, and occlusion invariance and in object-recognition performance.
Takeaways & Limitations
Contrastive self-supervised methods gain occlusion invariance from aggressive cropping but retain inferior viewpoint and category instance invariance relative to supervised models.
Takeaways & Limitations
The analysis focuses on the hypothesis that dataset bias explains the observed behavior, while representation capacity is identified as another possible factor.
Abstract
from arXiv · showhide
Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.
1 Introduction
Recent contrastive self-supervised methods gained strong downstream performance through instance discrimination, contrastive loss, and aggressive augmentation, but the encoded invariances and dataset dependence clarify both their strengths and limitations.
- Motivation: Earlier self-supervised pretext-task representations remained below supervised counterparts despite avoiding manual annotation.Examples include predicting future frames or image colors.
- Recent contrastive learning gains: Instance discrimination treats each image and its augmentations as one class while treating other images as negatives.Contrastive loss supplies the objective, and aggressive augmentation generates positive pairs from one image.
- Invariance analysis: Quantitative evaluation finds that recent self-supervised representations capture substantial occlusion invariance but lag supervised models in viewpoint and category instance invariance.These invariances are identified as important for object recognition.
- Augmentation and dataset bias: Aggressive cropping can pair non-overlapping regions from different object categories, making feature matching detrimental to downstream object recognition.The paper links this strategy’s success to object-centric biases in datasets such as ImageNet.
- Learning from videos: The paper proposes using naturally occurring transformations in videos to learn higher viewpoint invariance than image-based learning.The approach is reported to outperform MoCo-v2 trained on the same data across invariance measures and object-recognition tasks.
Representation Learning
Contrastive representation learning constructs positive pairs through transformations of the same sample and learns embeddings that bring positives together while separating negatives.
- Augmentations: Aggressive cropping can produce non-overlapping crops that depict different object categories, so matching their features may harm object recognition.The figure contrasts the original image with the resulting crop samples.
- Contrastive objective: Contrastive learning samples positive pairs whose representations should be similar and optimizes a normalized feature embedding.The positive-pair set is denoted D+.
- Contrastive objective: The objective uses a temperature parameter and a denominator that encourages discrimination of negative pairs outside D+.For large datasets, the negative-pair summation is approximated with K randomly chosen negatives per sample.
- Positive-pair construction: Self-supervised methods generate positives as two transformed versions of the same image and treat transformed versions of different images as negatives.This constructs D+ from transformation functions applied to each source sample.
- Augmentations: Common transformations include cropping, horizontal flipping, color jitter, grayscale conversion, and Gaussian blur.Composing these functions and varying their parameters controls the properties of the learned representation.
3 Related Work
Related work spans automated pretext tasks, instance discrimination, video-based supervision, and analyses that seek to explain why self-supervised representations succeed on downstream tasks.
- Self-supervised learning: Self-supervised representation learning is broadly classified into generative modeling and methods that learn features without human-annotated data.The learned representations are intended for multiple downstream tasks.
- Pretext tasks: Pretext tasks automatically construct supervision by predicting or reconstructing image content, including patch location, rotation, color, and missing regions.These tasks are designed to produce representations useful for object detection, classification, and segmentation.
- Instance discrimination: Instance discrimination treats an image and its transformed versions as one class, but its computational cost initially limited use with larger networks.Later work addressed this expense through more efficient approaches.
- SSL from videos: Video-based methods use tracked regions, motion segmentation, or temporal changes to learn viewpoint and deformation invariances.The paper builds on this line by tracking regions with weaker self-supervised features.
- Understanding representations: Downstream-task evaluation shows utility but does not explain why a representation works or how it generalizes to other tasks.The paper therefore studies encoded invariances and the role of the training dataset.
- Representation analysis: Prior analyses describe contrastive learning through alignment and uniformity, whereas this work focuses on invariances and training-dataset bias for object recognition.The paper argues that common augmentations rely on an object-centric training-dataset bias rather than explicitly enforcing task-relevant information retention.
4 Demystifying Contrastive SSL
The paper evaluates contrastive self-supervised representations through task-relevant invariances, then examines how augmentations and training-data structure shape those invariances and object discrimination.
- 4.1 Measuring Invariances: Task-relevant invariance is assessed by measuring whether representations remain stable under label-preserving transformations such as viewpoint, occlusion, illumination, and instance changes.The paper defines an ideal representation as invariant to transformations that preserve the ground-truth label.
- 4.1 Measuring Invariances: The invariance analysis uses hidden-unit firing rates over local transformation trajectories, normalizes target-conditioned rates by global firing rates, and aggregates the top-K units into RIS.A local trajectory contains transformed versions of a reference input, while RIS averages target-conditioned invariance over selected neurons and targets.
- 4.1 Measuring Invariances: Compared with supervised representations, MOCO and PIRL are better at occlusion invariance but weaker at viewpoint, instance, and instance-plus-viewpoint invariance.The paper attributes stronger occlusion invariance to aggressive cropping and the remaining gaps to instance discrimination’s treatment of instances as separate classes.
- 4.2 Augmentation and Dataset Biases: Random cropping can pair non-overlapping object parts as positives, encouraging similar representations for distinct objects and creating an object-discrimination mismatch.The paper illustrates this with scene crops containing objects such as chairs, tables, couches, refrigerators, and windows.
5 Learning from Videos
The paper leverages naturally occurring temporal transformations in videos to learn representations with stronger invariances and downstream discriminative performance than frame-only MOCO baselines.
- Motivation: Videos provide natural viewpoint, deformation, illumination, and occlusion transformations that image augmentations do not capture well.These transformations motivate using videos as training data for viewpoint-invariant representations.
- Baseline: The baseline applies MOCOv2 to uniformly sampled video frames, ignoring the temporal transformations occurring between frames.Three frames per video produce 118K training frames.
- Frame Temporal Invariance: Frame Temporal Invariance forms positive pairs from temporally separated frames and applies standard MOCO-v2 transformations independently.The resulting pairs treat naturally transformed frame views as positives for contrastive learning.
- Region Tracker: Region Tracker follows proposed regions across frames so temporally transformed regions, rather than entire scenes, supply contrastive positive pairs.Region matching uses feature distances between regions in subsequent frames, with an additional contrastive loss for positive and negative region pairs.
- Downstream Evaluation: Frame- and region-level temporal transformations improve discriminative power across Pascal, Pascal Cropped Boxes, Imagenet, and ADE20K.The evaluation covers image classification and semantic segmentation.
- Invariance Analysis: The proposed video representations demonstrate higher viewpoint and other invariances than Baseline MOCOv2, with Region Tracker comparable to full-Imagenet MOCOv2 using one-tenth as many samples.Region Tracker has significantly higher viewpoint and illumination-direction invariance than MOCOv2 trained on a 10% Imagenet subset.
6 Conclusion
The paper evaluates why contrastive self-supervised representations work, identifies their learned invariances and dataset dependence, and proposes video-based temporal transformations as an alternative.
- Conclusion: The framework shows that aggressive cropping produces occlusion invariance while relying heavily on an object-centric dataset bias.Compared with supervised models, the representations have inferior viewpoint, illumination-direction, and category-instance invariances.
- Conclusion: The proposed strategy improves representation invariances by leveraging naturally occurring temporal transformations in videos.This alternative targets transformations available in unstructured video data.
A Comparison of Invariance Measure to Goodfellow et. al[26]
The paper modifies an existing invariance measure to account for task labels and the discriminative power required by class-based tasks.
- Motivation: The revised measure adapts an invariance framework so scores reflect both encoded invariance and representation discriminative power.The original method focuses purely on invariance and can favor non-discriminative representations.
- Chosen Thresholds: Class-dependent adaptive thresholds set each hidden unit’s global firing rate to the fraction of samples in the corresponding class.This lets a hidden unit fire on all samples belonging to a class rather than only a fixed global fraction.
- Local Firing Rate: Separate local firing rates are computed for each class, assigning equal importance to classes in imbalanced datasets.A single rate across all classes would weight larger classes more heavily.
- Invariance Scores: Task-dependent invariance scores are computed as I_y(i) = L_y(i)/G_y(i), with Top-K hidden units selected separately for each class.The mean task-dependent invariance score is then computed.
- Comparison: Selecting Top-K hidden units across all classes can penalize units that are optimally discriminative and invariant for particular classes.The paper argues that class-specific scoring is essential for class-discriminative tasks.
B Implementation Details: Learning from Videos
The video-based method trains MOCOv2 on temporally separated frame pairs, optionally using tracked patches, with a queue-based contrastive-learning procedure. It uses TrackingNet video chunks, selective-search proposals, recursive track scoring, and ResNet-50 features.
- Dataset Creation: TrackingNet provides 30K video sequences, each split into four maximally spaced chunks of 60 consecutive frames for training.Each extracted chunk is treated as a separate video.
- Training Procedure: When tracking is enabled, the algorithm samples a patch pair tracked across frames and computes its loss and keys alongside the full-frame inputs.The tracked patch keys are also enqueued in the contrastive dictionary.
- Generating Tracks: Selective search generates region proposals, and tracks are retained only when their recursive cosine-similarity score exceeds a chosen threshold.The top 300 proposals are used when a frame produces more than 300 regions; region features come from layer-3 ROI pooling.
- Sampling Frames: The method samples frame pairs separated by at least k = 40 frames to train frame temporal invariance.Each pair is drawn from temporally separated frames within the video sequences.
- Implementation Details: The implementation follows MOCOv2 with ResNet-50, query-network SGD updates, momentum key-network updates, and a queue storing K keys.The queue is updated by enqueuing the current minibatch and dequeuing the earliest minibatch.