Intriguing Properties of Contrastive Losses

Ting Chen; Calvin Luo; Lala Li

Source-linked AI summary

Intriguing Properties of Contrastive Losses

Ting Chen, Calvin Luo, Lala Li

arXiv:2011.02803v3Nov 5, 2020cs.LGcs.AIcs.CVstat.ML

TL;DR

Contrastive learning leaves open how loss design, global instance objectives, and competing shared features affect learned representations. The paper generalizes contrastive losses, tests global objectives on multi-object images, and constructs controllable datasets to study suppression. It finds similar losses with deep projection heads, meaningful local features from global objectives, and suppression of difficult features by a few easy shared bits, while identifying feature suppression as an open challenge.

Problem
The paper asks how contrastive loss design, global instance objectives, and competing features shared across augmented views affect representation learning.
Method
The authors generalize the contrastive loss, evaluate instance-based learning on multi-object images, and construct datasets with explicit, controllable competing features.
Results
The generalized losses differ little with deep projection heads, global objectives learn meaningful local features on multi-object images, and easy shared features can suppress other competing features.
Takeaways & Limitations
Feature suppression remains an open challenge and can bottleneck contrastive learning when existing augmentations cannot fully address it.
Takeaways & Limitations
The study focuses mostly on contrastive learning with explicit negatives, and many proposed image datasets are not fully realistic.

Abstract

from arXiv · show

We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.

1 Introduction

The paper examines three properties of contrastive learning: generalized losses, global instance objectives on multi-object images, and suppression among competing shared features.

Existing methods rely critically on hand-crafted data augmentation to favor some competing feature sets over others.When augmentation cannot fully address suppression, contrastive learning may face a bottleneck and saturate at an unsatisfactory level.
The authors generalize the standard contrastive loss and find that differences between instantiations are small with a deep projection head.The generalized loss broadens the family of objectives studied while preserving similar performance under this architectural condition.
Instance-based contrastive objectives can learn from images containing multiple objects and produce meaningful local features despite operating on global image representations.The objective encodes each image into a single vector, yet the learned representation can retain hierarchical local information.
The study constructs datasets with explicit and controllable competing features to systematically investigate feature suppression in contrastive learning.The datasets are designed to isolate and vary competing features shared across augmented views.
A few bits of easy-to-learn shared features can suppress, and even fully prevent, learning other competing feature sets.With multiple objects, the dominant object can suppress learning of smaller objects.

2 Generalized contrastive loss and differences among its instantiations

The paper generalizes contrastive loss into alignment and distribution-matching components, then compares alternative instantiations using linear evaluation. These variants produce similar representations when training uses a deep non-linear projection head.

Generalized loss formulation: The generalized contrastive loss combines alignment of augmented views with distribution matching against a high-entropy prior.The standard contrastive loss is presented as a special case of this decomposition.
Generalized loss formulation: Sliced Wasserstein Distance enables distribution matching for diverse priors beyond the uniform hypersphere.The method projects samples into random orthogonal one-dimensional subspaces and sums their Wasserstein distances.
Empirical comparison: Linear evaluation shows little difference among generalized contrastive-loss variants on CIFAR-10, especially after more than 200 training epochs.The comparison follows SimCLR settings and uses linear evaluation.
Empirical comparison: On ImageNet, discrepancies among losses disappear when a 3-layer non-linear projection head is used.The figure reports linear evaluation accuracy for ResNet-50 models trained with different losses and projection-head depths.
Empirical comparison: A deep projection head also reduces the effect of batch size on representation quality when learning-rate scaling is applied.The paper reports this phenomenon for standard contrastive loss and notes additional results for other losses in Appendix A.3.

3 Instance-based objective can learn on images with multiple objects and learn good local features

Instance-based contrastive learning can learn useful representations from images containing multiple objects, including meaningful local features despite using global image representations. Performance remains stable with moderate object counts, while overlap and later-layer structure reveal important limits and hierarchy.

3.1 SimCLR can learn on images with multiple objects: Increasing overlap between digits makes contrastive learning from multiple objects harder, as shown by the growing gap between random and in-grid placement.Random placement permits overlapping digits, whereas in-grid placement prevents overlap.
3.1 SimCLR can learn on images with multiple objects: SimCLR maintains top-1 linear-evaluation accuracy as the number of digits increases to eight, regardless of placement strategy.Evaluation uses images containing a single digit, while training varies the number of digits from 1 to 16.
3.2 SimCLR learns local features that exhibit hierarchical properties: K-means visualizations show SimCLR and supervised features grouping regions by object parts, unlike raw pixel features.Later network layers contain more object- and part-related features, while earlier layers contain more edge-related features.
3.2 SimCLR learns local features that exhibit hierarchical properties: On COCO images, SimCLR and supervised local-feature groupings tend to overlap with object classes.The comparison includes raw RGB patch features as a baseline.

4 Feature suppression limits the potential of contrastive learning

The paper constructs controllable datasets to study competing shared features and finds that easy-to-learn features can suppress harder ones. This suppression affects object recognition, smaller objects, and representations across several datasets and contrastive configurations.

4 Feature suppression limits the potential of contrastive learning: Contrastive methods critically rely on data augmentation to remove or favor competing features, creating a saturation challenge when augmentations cannot address suppression.The paper connects this challenge to the need for exponentially larger batch sizes as the number of competing bits increases.
4.1 Datasets with explicit and controllable competing features: The authors construct three datasets with controllable competing features using channel addition or channel concatenation.The datasets include DigitOnImageNet, MultiDigits with varying digit sizes, and RandBit.
4.2 Competing features suppress each other: Increasing unique MNIST digits improves MNIST linear-evaluation accuracy while dramatically reducing ImageNet accuracy for SimCLR features.A supervised ResNet baseline shows little impact of the number of unique MNIST digits on ImageNet top-1 accuracy.
4.3 The presence of dominant object suppresses the learning of features of smaller objects: For SimCLR, enlarging one digit causes the smaller digit’s representation to degenerate nearly to random-network performance, while the dominant digit remains above 99% accuracy.Temperature tuning reduces suppression somewhat but does not change the trend.
4.4 Extra channels with a few bits of easy-to-learn mutual information suppress the learning of all features in RGB channels: A few added random bits rapidly reduce linear-evaluation accuracy, with the effect persisting on CIFAR-10 and ImageNet.The effect is not avoided by changing contrastive losses, batch sizes, or momentum-contrast memory mechanisms.

5 Related Work

Related work has studied contrastive learning theoretically, empirically, through alignment and uniformity, and through mutual-information connections. The paper identifies feature suppression as a comparatively underexplored problem and presents its systematic quantitative study as novel.

Related work: Earlier work showed that alignment-and-uniformity formulations and other distribution-matching losses can achieve similar performance to standard contrastive loss.This paper further generalizes that factorization.
Related work: Contrastive loss has been connected to mutual information, although higher mutual information does not guarantee better representation quality.The passage also notes limitations of mutual-information estimation.
Related work: Prior work studies contrastive learning through theoretical and empirical analyses.The cited literature includes multiple directions for understanding contrastive objectives.
Related work: The authors state that feature suppression had received little prior study and that they are the first to investigate it quantitatively and systematically.They describe the topic as an open question for future investigation.

6 Conclusion

The study identifies feature suppression as an open challenge and a bottleneck when augmentation cannot fully resolve competing features. It also acknowledges scope and realism limitations in the experiments.

Feature suppression remains an open challenge in contrastive learning and can bottleneck learning when augmentation cannot fully address it.
The study focuses mostly on contrastive methods with explicit negatives, leaving clustering-based and negative-free methods for future work.
Many proposed image datasets are not fully realistic because controllable studies of competing features and multiple objects are difficult on realistic large-scale datasets.

A.1 Experimental setup

The experiments use standard contrastive-learning architectures, augmentations, optimization, and linear evaluation, while tuning loss-specific hyperparameters for fair comparisons.

ResNet-50 with a 2-layer projection head is the default architecture, with output dimensionality 64 for CIFAR-10 and 128 for ImageNet.
The experiments use square-root learning-rate scaling with batch size and a LARS optimizer, while training epochs and batch sizes vary by experiment.
Representation quality is measured by the accuracy of a trained linear classifier on the learned features.
Hyperparameters are optimized separately for NT-Xent, decoupled NT-Xent, SWD-based, and alignment losses.

A.2 Temperature τ is (within a range) inversely correlated to weighting λ of distribution loss

The appendix separates temperature and distribution-loss weighting to study their effects on representation distributions and evaluation performance. Across the examined range, the two parameters are generally inversely correlated.

Random 1-D projections should appear Gaussian-like when learned representations match Gaussian or uniform-hypersphere priors.
Temperature τ and weighting λ both control distribution matching, with higher temperature producing an effect similar to larger distribution-loss weighting.
For SWD, projections become more Gaussian as λ increases; for NT-Xent, they become more Gaussian as τ decreases.
The generalized loss decouples temperature’s distribution-loss scaling from its Gaussian-kernel width, allowing their effects to be studied separately.
In linear evaluation of ResNet-18 trained for 200 epochs, τ and λ are inversely correlated across most of the examined range, so one can be fixed while tuning the other.

A.3 Linear evaluation of generalized contrastive losses on CIFAR-10 and ImageNet

Linear evaluation compares ResNet-50 representations trained with different generalized contrastive losses on CIFAR-10 and ImageNet, including the effect of batch size. The reported differences are small at reasonable batch sizes.

Linear evaluation measures top-1 accuracy for ResNet-50 trained with different losses on CIFAR-10 and ImageNet.
Differences between losses are small with reasonable batch sizes, including 128 for CIFAR-10 and 1024 for ImageNet.
The comparisons use square-root learning-rate scaling and include results across different batch sizes.

B.1 Extra results on CIFAR-10 and ImageNet with random bits added

Adding only a few random or competing-feature bits sharply degrades contrastive representations on CIFAR-10 and similarly affects ImageNet. This degradation persists across contrastive losses and training configurations.

ImageNet: The appendix compares linear evaluation across different losses and projection-head configurations on ImageNet.
CIFAR-10: A few extra competing-feature bits quickly reduce CIFAR-10 linear-evaluation accuracy.The reported bits are calculated from the total size of a uniform integer distribution and therefore overestimate actual bits because of collisions.
CIFAR-10: Less than 15 competing-feature bits degrade representation quality to the level where RGB channels are completely ignored.
CIFAR-10: The detrimental effect cannot be avoided by changing contrastive losses, batch sizes, or the momentum-contrast memory mechanism.Smaller temperature or larger weighting slightly mitigate degeneration, but their no-extra-bits baseline performance is also worse.
ImageNet: Similar feature-suppression results are observed on ImageNet.The cited appendix figure evaluates learned features when a few bits of competing features are added.

B.2 Distribution matching loss, LogSumExp or SWD, saturates with a few bits of entropy

Distribution-matching losses saturate after only a few bits of dataset entropy. Increasing representational bits then requires rapidly growing batch sizes, limiting scalability.

Distribution-matching loss saturates quickly with only a few bits of entropy.The experiment uses square images with k binary channels and evaluates LogSumExp or sliced Wasserstein distance without the alignment term.
Temperature and batch size affect the saturation behavior of distribution-matching losses.
Linear growth in representation bits requires exponential growth in batch size.Thirty bits would require more than 1 billion samples in a batch, exceeding practical dataset sizes.