Source-linked AI summary
Representation Learning by Learning to Count
Mehdi Noroozi, Hamed Pirsiavash, Paolo Favaro
TL;DR
The paper addresses representation learning from unlabeled images without manual annotation. It learns features that satisfy counting-based relationships under image transformations, using downsampling and tiling with a contrastive loss. The learned representations perform on par with or exceed state-of-the-art transfer-learning methods, while the counting scale is not directly controlled.
Problem
Learning discriminative representations for semantic image understanding without costly, time-consuming, and error-prone manual annotation remains a central challenge.
Method
The method uses counting visual primitives as an artificial supervision signal, relating downsampled images and tiles through a feature constraint used to train a neural network with contrastive loss.
Results
The learned features perform on par with or exceed state-of-the-art methods on standard transfer-learning benchmarks.
Takeaways & Limitations
The framework learns representations from unlabeled data and can be extended to other tasks and transformations or combined with partially labeled data in semi-supervised learning.
Takeaways & Limitations
The method provides no simple way to control whether its counting features represent object parts, whole objects, object groups, or combinations thereof.
Abstract
from arXiv · showhide
We introduce a novel method for representation learning that uses an artificial supervision signal based on counting visual primitives. This supervision signal is obtained from an equivariance relation, which does not require any manual annotation. We relate transformations of images to transformations of the representations. More specifically, we look for the representation that satisfies such relation rather than the transformations that match a given representation. In this paper, we use two image transformations in the context of counting: scaling and tiling. The first transformation exploits the fact that the number of visual primitives should be invariant to scale. The second transformation allows us to equate the total number of visual primitives in each tile to that in the whole image. These two transformations are combined in one constraint and used to train a neural network with a contrastive loss. The proposed task produces representations that perform on par or exceed the state of the art in transfer learning benchmarks.
1. Introduction
The paper proposes counting visual primitives as a self-supervised pretext task for learning representations without manual annotation. It uses image transformations and their counting relationships, with experiments reporting state-of-the-art transfer performance.
- Manual annotation for semantic image understanding is costly, time-consuming, and prone to errors, motivating scalable self-supervised alternatives.
- Counting visual primitives supplies an artificial supervision signal because primitives in non-overlapping tiles should sum to those in the whole image.The relationship is illustrated by the tiled regions in Figure 1.
- The proposed pretext task is hypothesized to encourage representations that disentangle high-level factors of variation through a simple arithmetic relation.
- The framework fixes image transformations and learns a representation satisfying the corresponding feature-space relationships, with possible extensions beyond counting, scaling, and tiling.
- The learned representations perform on par with or exceed state-of-the-art methods on standard transfer learning benchmarks.
2. Prior Work
Prior self-supervised methods derive surrogate labels from information already present in unlabeled images, whereas this paper introduces counting relationships between related image parts. The method uses these relationships without object-count annotations.
- This paper defines a binary relationship for image pairs based on whether they contain the same number of visual primitives.The relationship is expected for identical images under certain transformations but generally not for different images.
- Unlike methods using relationships between different tracked images or labels revealing cross-image relationships, the approach relates different parts of the same image.
- Counting is used as a representation-learning pretext task rather than as the primary object-counting problem.Training does not use labels specifying the number of objects.
3. Transforming Images to Transform Features
The method learns a feature representation whose outputs obey a prescribed relationship under a family of image transformations. For counting, downsampling and four-image tiling are combined so tile counts relate arithmetically to the downsampled image count.
- A feature φ maps transformed images to k-dimensional vectors, while a feature transformation g combines the resulting J features.
- The representation is learned by enforcing g(φ(G1 ◦ x), ..., φ(GJ ◦ x)) = 0 for every image x.
- The counting transformation family uses downsampling by a factor of 2 and four tiles extracted from a 2 × 2 grid.These transformations produce images of the same size.
- The resulting relationship is used as the main building block for learning features that count visual primitives.
- Unlike standard equivariance, which characterizes a given feature, this approach fixes transformations and searches for a representation satisfying them.
- The method does not directly control the scale of counted content, which may range from object parts to whole objects or groups.The scale can depend on the counting-vector size, loss function, and training data.
4. Learning to Count
The method trains an AlexNet-based representation with a contrastive counting objective using tiled and downsampled image inputs. Despite sparse features, the learned hidden representations transfer well across classification, detection, and segmentation.
- A convolutional network is trained with an l2-based counting objective, but the objective admits the trivial all-zero representation.The method therefore adds a contrastive loss to avoid collapse.
- The architecture uses tied siamese branches to process four tiles, a downsampled image, and a randomly chosen downsampled image for contrastive training.Because the features are shared, this corresponds to a 6-way siamese network.
- The contrastive term requires counting features from different images to differ, counteracting the bias toward counting as few visual primitives as possible.This tradeoff encourages features that both satisfy the counting constraint and distinguish image content.
- 30 nonzero entries in the ImageNet-trained representation coexist with strong transfer performance on classification, detection, and segmentation tasks.The reported transfer results concern hidden representations despite their sparsity.
5. Experiments
Experiments evaluate the learned counting representation through transfer learning, ablations, and analyses of what its features respond to. It performs competitively across classification, detection, and segmentation while revealing sparse, semantically structured activations and sensitivity to shortcut cues.
- Transfer-learning benchmarks: The representation is evaluated on PASCAL classification, detection, and segmentation, plus ImageNet classification, using standard transfer-learning protocols.PASCAL classification and detection use VOC 2007, while segmentation uses VOC 2012; ImageNet evaluation uses a linear classifier on frozen layers.
- Transfer-learning benchmarks: On PASCAL, the method either outperforms previous methods or achieves the second-best performance across classification, detection, and segmentation.Classification and detection are measured by mAP, while segmentation is measured by mIoU.
- Linear classification: The method matches state-of-the-art performance on ImageNet and improves significantly on Places, including state-of-the-art results with conv1–conv4 features on Places.The Places setup tests cross-dataset generalization by pretraining on ImageNet without labels and evaluating frozen features on Places.
- Ablation studies: Ablations vary interpolation methods, training size and color, and counting-vector dimensionality to compare their effects on PASCAL VOC 2007 detection.The study also identifies trivial solutions and develops countermeasures against shortcuts that could produce poor representations.
- Ablation studies: The network can learn downsampling style, with pairwise errors and their test-time variability reported across training and testing interpolation methods.The analysis identifies similar artifacts for linear/area and cubic/Lanczos downsampling and discusses chromatic aberration as a possible cue.
- Representation analysis: Feature magnitudes distinguish texture-only images from images containing multiple object instances or large objects, while neurons cluster images with similar scene layout and content.Cropped-image analysis indicates that the feature is more sensitive to composite images than to low-level texture.
6. Conclusions
The method learns representations without annotated data by using counting as a pretext constraint trained with contrastive loss. The learned features capture semantic content, organize related scenes, and outperform prior state-of-the-art transfer-learning methods.
- The method learns representations without relying on annotated data.
- Counting is formalized as a constraint relating visual primitives counted in image tiles to those counted in a downsampled image.
- A contrastive loss trains the neural network using this counting constraint.
- The learned features count non-trivial semantic content and qualitatively cluster images with similar scene outlines.
- The learned representations outperform previous state-of-the-art methods on transfer-learning benchmarks.