Learning Representations by Predicting Bags of Visual Words

1. Introduction

• Pixel-reconstruction tasks can be difficult and need not teach image-understanding skills; the introduction contrasts continuous, perturbation-sensitive pixels with NLP's discrete words.The paper frames the open question as which kind of self-supervision to use for learning useful image representations.Source: §1 · p. 1
• The method densely quantizes a self-supervised convnet's feature maps with a k-means vocabulary, assigning discrete visual words to image locations.These codes form spatially dense image descriptions that can serve as discrete prediction targets.Source: §1 · pp. 1–2
• A second convnet receives a perturbed image and predicts the original image's Bag-of-Words histogram, avoiding pixel-level reconstruction targets.The figure describes the original-image BoW vectors as ground truth for the prediction stage.Source: §1 · p. 2, Figure 1 · p. 2
• The simple design avoids adding computationally intensive attention or PixelCNN-like components and can be incorporated into other learning problems.The authors cite few-shot learning, semi-supervised learning, and unsupervised domain adaptation as possible applications.Source: §1 · pp. 2–3
• BoWNet representations outperform the initial self-supervised model, with better VOC detection and comparable Places205 classification on unseen classes than supervised representations.The authors describe the learned representations as high quality and report better generalization on detection and similar generalization on unseen Places205 classes.Source: §1 · p. 3

2.1. Building spatially dense discrete descriptions

• A frozen self-supervised convnet assigns local features to nearest k-means vocabulary centroids, producing spatially dense visual-word descriptions.The vocabulary centroids are learned from feature maps extracted from the unlabeled image set.Source: §2.1 · p. 3, §2 · p. 3

2.2. Generating Bag-of-Words representations

• The K-dimensional BoW target records visual-word counts or binary presence, then L1-normalizes them into a sparse probability distribution.ImageNet uses binary targets, while CIFAR-100 and MiniImageNet use histograms.Source: §2.2 · p. 3, §2.3 · p. 3

2.3. Learning to “reconstruct” BoW

• A randomly initialized second convnet predicts the clean image’s BoW from a perturbed view through a linear-softmax head trained with cross-entropy.The first convnet builds targets rather than initializing the second model’s parameters.Source: §2.3 · pp. 3–4
• Color and geometric perturbations require robust features and prediction of words in cropped-away regions; CutMix mixes images and area-weighted BoW targets to infer missing and swapped regions.CutMix’s target is the convex combination of the two images’ BoW targets, weighted by patch-over-image area.Source: §2.3 · p. 4

2. Approach

• The trained model can replace the target-generating convnet for another training round, although the first iteration already achieves strong results.The paper suggests only a few additional iterations may be applied.Source: §2 · p. 3, §2.3 · p. 4

3. Related Work

• Unlike NetVLAD [2], which learns a vocabulary and soft-quantizes activations, this method trains with self-supervision to predict BoW directly, bypassing quantization and aggregation.Source: §3 · p. 4
• Rather than reconstructing perturbed input content, the method predicts the clean image’s BoW, enabling learning non-trivial correlations between local visual patterns.Source: §3 · p. 5

3.1. Discussion

• Compared with Deep(er) Clustering [9] [8], which assigns global image labels, spatial BoW encodes multiple local concepts and encourages localized patterns and contextual prediction of missing regions.Source: §3.1 · p. 5
• For urban scenes where images differ by few details, the authors argue that local statistics may be more appropriate than recognition under strong perturbations.Source: §3.1 · p. 5

4.1.1 Implementation details

• CIFAR-100 uses WRN-28-10 with a RotNet-derived K=2048 vocabulary, while MiniImageNet uses WRN-28-4 trained on its 64 training classes.For CIFAR-100, the vocabulary is built from the last residual block of RotNet.Source: §4.1.1 · p. 5

4.1.2 Evaluation protocols.

• Evaluation freezes representations for linear classification and episodic few-shot tasks; MiniImageNet few-shot evaluation uses test classes unseen during self-supervised training.CIFAR-100 few-shot evaluation samples 5-way tasks with 1, 5, 10, or 50 examples per class.Source: §4.1.2 · p. 6

4.1.3 Results

• BoWNet improves every CIFAR-100 and MiniImageNet evaluation metric over RotNet by at least 10 percentage points; iteration improves results further except in 1-shot.On MiniImageNet novel classes, BoWNet is close to supervised Cosine Classifier performance in the 10-shot and especially 50-shot settings.Source: §4.1.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6
• Deeper Clustering trails BoWNet, while adding spatial-pyramid information does not improve representations.CutMix further improves representation quality. RelLoc initialization matches or exceeds RotNet-based BoWNet, whereas random initialization lowers performance but remains above RotNet and RelLoc.Source: §4.1.3 · pp. 6–7, Table 1 · p. 6, Table 3 · p. 6

4.2. Self-supervised training on ImageNet

• BoWNet outperforms prior self-supervised methods by a significant margin on frozen-feature linear classification for ImageNet and Places205; on Places205, its ImageNet-trained pool5 representation is only 0.9 points below supervised representations.Source: §4.2 · p. 8
• On VOC detection, BoWNet outperforms supervised ImageNet pretraining under the same fine-tuning conditions, and conv5 beats conv4 on all metrics.The table reports AP50, AP75, and APall on test07; BoWNet scores are averaged over three trials.Source: §4.2 · p. 8, Table 6 · p. 8

5. Conclusion

• BoWNet uses spatially dense visual-word descriptions as self-supervised targets and, despite relying on labels from a standard self-supervised model, outperforms that initial model.The authors attribute more discriminative learning of local visual patterns to reconstructing BoW vectors from perturbed images and discretizing the output space.Source: §5 · p. 8

A.1. Visualizing the word clusters

• Visual-word clusters correspond to mid-to-higher-level visual concepts, illustrated by patches nearest each cluster centroid.The visualization shows 16 patches per cluster, selected by Euclidean distance to its centroid.Source: §A.1 · p. 12

A.2. t-SNE scatter plots of the learnt self-supervised

• On CIFAR-100, the learned features form class-specific clusters in a t-SNE plot of 20 randomly selected classes.Each point represents an image feature and is colored by its class.Source: §A.2 · p. 12, Figure 2 · p. 12

Impact of vocabulary size

• Increasing vocabulary size from K = 256 to K = 2048 improves CIFAR-100 linear classification, while K = 4096 adds no further improvement.The table uses an initial method version with less aggressive augmentations.Source: §B.1 · p. 12, Table 7 · p. 12

Reparametrized linear layer for Ω(·)

• Using a standard linear layer instead of the reparametrized Ω(·) layer deteriorates performance, especially on linear classification.The reparametrization L2-normalizes word weights and applies one learnable magnitude, avoiding per-word magnitudes that can favor frequent words.Source: §B.1 · p. 12, Table 8 · p. 12

Predicting y(˜x) instead of y(x)

• Predicting the perturbed image's BoW instead of the original image's BoW causes a significant performance drop.Predicting the original target is intended to encourage perturbation-invariant and context-aware features.Source: §B.1 · pp. 12, 14, Table 8 · p. 12

Histogram BoW vs Binary BoW

• Histogram BoW and Binary BoW achieve very similar linear-classification and few-shot performance.Histogram BoW counts each visual word's occurrences; Binary BoW records whether it appears.Source: Table 8 · p. 12, §B.1 · p. 14

Experimental setting

• The small-scale ImageNet experiments use ResNet18, 20% of training images, and only a few epochs.Representations are frozen and evaluated with 1000-way linear classifiers on the last residual block's feature map.Source: §B.2 · p. 14

Results

• Increasing vocabulary size from K = 2048 to K = 20000 significantly improves linear-classifier performance, while among BoW representations built from conv3, conv4, and conv5 feature blocks, conv4-based BoW performs best.Source: §B.2 · p. 14
• The vocabulary-size pattern differs by dataset: CIFAR-100 showed no improvement beyond K = 2048, suggesting the optimal size depends on dataset complexity.The ImageNet comparison is from a small-scale experiment.Source: §B.2 · p. 14

• Full experimental results for the method on ImageNet and Places205 classification are provided in Table 10.Source: §B.3 · p. 14

C. Implementation details

• The rotation head uses four residual layers for WRN-28-10 and WRN-28-4, but one for ResNet-50.The corresponding input and output channel counts are 640, 256, and 2048, respectively.Source: §C.1 · p. 14
• Training predicts one of four rotations (0°, 90°, 180°, or 270°) for each rotated image, using cross-entropy over the four classes.Source: §C.1 · p. 14
• After training, the rotation prediction module is discarded, and only the feature extractor is used for subsequent stages.Source: §C.1 · p. 14

C.2. Building BoW for self-supervised training

• ImageNet BoW targets use visual words from both the original and horizontally flipped image; targets are pre-cached, and edge feature vectors are excluded in all experiments.Source: §C.2 · p. 14

C.3. Few-shot protocol

• MiniImageNet experiments use the typical few-shot protocol: train on base classes, then evaluate on different novel classes, with each class set split into train and validation subsets; the dataset has 64 training classes.Source: §C.3 · p. 14
• For CIFAR-100, evaluation samples from test images of 20 selected classes, while self-supervised training uses training images from all 100 classes.Source: §C.3 · p. 15
• Each task samples five novel classes and 15 test images per class, with 1, 5, 10, or 50 training examples per class; performance averages over 2,000 tasks.Source: §C.3 · p. 15