Source-linked AI summary
Local Aggregation for Unsupervised Learning of Visual Embeddings
Chengxu Zhuang, Alex Lin Zhai, Daniel Yamins
TL;DR
Large-scale visual recognition still needs unsupervised methods that approach supervised performance without costly annotations. The paper introduces Local Aggregation, which learns dynamically clustered embeddings balancing similarity-based grouping with separation, and reports state-of-the-art transfer results across ImageNet, Places205, and PASCAL VOC.
Problem
Unsupervised visual-learning methods aim to avoid costly annotation, but their performance has typically lagged supervised networks in large-scale visual recognition.
Method
Local Aggregation iteratively identifies embedding-space neighbors and optimizes a neural embedding to bring similar inputs together while separating dissimilar ones.
Results
LA achieves state-of-the-art unsupervised transfer learning on ImageNet classification, Places205 scene recognition, and PASCAL VOC object detection.
Takeaways & Limitations
Local Aggregation produces visual representations that support transfer learning across several challenging recognition tasks.
Takeaways & Limitations
The method’s clustering choice is constrained by scalability, and replacing k-means with DBSCAN produces trivial representations in the studied high-dimensional, variable-density setting.
Abstract
from arXiv · showhide
Unsupervised approaches to learning in neural networks are of substantial interest for furthering artificial intelligence, both because they would enable the training of networks without the need for large numbers of expensive annotations, and because they would be better models of the kind of general-purpose learning deployed by humans. However, unsupervised networks have long lagged behind the performance of their supervised counterparts, especially in the domain of large-scale visual recognition. Recent developments in training deep convolutional embeddings to maximize non-parametric instance separation and clustering objectives have shown promise in closing this gap. Here, we describe a method that trains an embedding function to maximize a metric of local aggregation, causing similar data instances to move together in the embedding space, while allowing dissimilar instances to separate. This aggregation metric is dynamic, allowing soft clusters of different scales to emerge. We evaluate our procedure on several large-scale visual recognition datasets, achieving state-of-the-art unsupervised transfer learning performance on object recognition in ImageNet, scene recognition in Places 205, and object detection in PASCAL VOC.
1. Introduction
Unsupervised learning could reduce reliance on costly annotations, but its visual recognition performance has typically lagged supervised methods. Local Aggregation addresses this gap by learning dynamically clustered embeddings that support strong transfer across visual tasks.
- Motivation: Unsupervised DCNN training is sought because unlabeled data are often readily available, whereas supervised methods require costly large-scale annotation.
- Motivation: Infants can group perceptually similar novel stimuli before developing explicit object-category concepts, motivating unsupervised learning based on visual similarity.
- Method: Local Aggregation iteratively identifies close neighbors in a latent embedding space and optimizes the network to strengthen local aggregation.
- Method: The method separates naturally dissimilar inputs while allowing statistically similar inputs to form emergent soft clusters of different scales.
- Results: 60.2% top-1 accuracy on ImageNet classification was achieved by an unsupervised LA model, surpassing the supervised AlexNet milestone.
- Results: LA-trained models achieved state-of-the-art unsupervised transfer learning on PASCAL object detection after further fine-tuning.
2. Related Work
Prior unsupervised representation-learning methods include iterative clustering, instance separation, missing-data prediction, and generative modeling. Local Aggregation is related to clustering and instance recognition but combines local separation with clustering rather than enforcing either globally.
- Overview: Unsupervised representation learning spans iterative clustering, instance recognition, self-supervised missing-data tasks, and generative models.
- Clustering methods: DeepCluster alternates feature clustering with using cluster assignments as supervision labels for subsequent DCNN training.
- Instance separation: Instance Recognition treats each example as its own category and optimizes embeddings in which all examples are well separated.
- Local Aggregation: Local Aggregation differs from Instance Recognition by balancing separation and clustering on a per-example basis through a local aggregation criterion.
- Self-supervised tasks: Missing-data tasks learn representations by hiding input information and predicting it from the remaining visible information.
- Generative models: Generative models reconstruct input images from bottlenecked latent representations, whose downstream power for object recognition remains incompletely realized.
3. Methods
Local Aggregation learns a compact embedding by dynamically contrasting close neighbors with background neighbors. Its softmax-based metric encourages local clusters while separating non-close points, with a memory bank making the procedure tractable at scale.
- Embedding objective: LA embeds images into a compact D-dimensional space where similar images cluster and dissimilar images separate.The embedding function is realized by a neural network and optimized iteratively.
- Local aggregation metric: The local aggregation metric compares closeness within close neighbors against background neighbors and is maximized by tuning the embedding network.The metric is formulated using a non-parametric softmax-based probability and optimized through the loss defined for each image.
- Neighbor identification: Background neighbors are the k closest embedded points under cosine distance, while close neighbors are obtained from clustering representations into groups.The number k of background neighbors and the clustering parameters are hyperparameters.
- Neighbor identification: Multiple clusterings are aggregated to make close-neighbor identification more stable despite clustering noise and arbitrariness.The procedure uses H distinct clusterings and aggregates neighbors across them.
- Local aggregation metric: Minimizing the loss separates non-close background neighbors while encouraging movement toward close-neighbor sub-categories, producing clusters at natural scales.The softmax scale parameter τ controls the probability formulation, and the loss includes λ∥θ∥2 regularization.
- Efficient implementation: A memory bank stores running-average feature vectors so neighbor identification and metric computation need not recompute all embeddings at every step.The bank is initialized with random D-dimensional unit vectors and updated by mixing stored and current features.
4. Results
The experiments evaluate Local Aggregation across image classification, scene categorization, and object detection, finding strong transfer-learning performance across architectures. LA performs especially well with deeper networks, while further hyper-parameter search may improve results.
- Experiment Settings: Most training parameters follow prior work, and the authors note that further hyper-parameter search could improve results because LA uses a substantially different optimization goal.The reported settings include τ = 0.07, D = 128, λ = 0.0001, and t = 0.5.
- Evaluation Protocol: The evaluation transfers ImageNet-trained representations to held-out ImageNet classification, Places scene classification, and PASCAL VOC 2007 detection.Classification experiments include both linear readouts and K-nearest-neighbor evaluation.
- Object Recognition: LA-trained models significantly outperform other unsupervised methods on ImageNet classification, especially with deeper architectures.ResNet-50 reaches 60.2% top-1 accuracy, surpassing AlexNet trained directly with supervision.
- Scene Categorization: LA surpasses previous methods on Places transfer learning across all architectures, with larger gains for deeper networks.The authors interpret these results as evidence of strong generalization of the learned visual representations.
- Object Detection: LA achieves state-of-the-art unsupervised transfer learning for PASCAL VOC object detection.The performance gap between the best unsupervised methods and supervised controls is comparatively smaller for detection than for classification.
5. Analysis
The analysis examines how LA shapes embedding density, nearest-neighbor behavior, class structure, and sensitivity to key design choices. It finds locally dense but globally sparse representations, coherent grouping, and benefits from dynamic neighbors, robust clustering, and capacity-matched parameters.
- Density distribution: LA embeddings are locally dense while remaining generally sparse across the space.The objective minimizes distances to close neighbors and maximizes distances to background neighbors.
- Density distribution: LA produces higher local density than Instance Recognition, while its background density is only slightly higher.The comparison uses ResNet-18 features trained with LA and Instance Recognition.
- Density distribution: Deeper ResNet-50 embeddings show higher local density and lower background density than shallower ResNet-18 embeddings.This confirms the expectation associated with lower minimums of the LA objective in deeper networks.
- Success and failure examples: Nearest-neighbor examples show that LA groups same-category images across backgrounds and viewpoints while also organizing salient visual context.Failure cases often reflect ambiguous ImageNet labels, such as distinguishing humans playing trombones from trombones themselves.
- Ablations: Dynamic nearest-neighbor background selection outperforms using all inputs or a fixed clustering procedure.The ablations isolate the role of dynamically rescaling the local background against which closeness is judged.
- Ablations: Cluster-based close-neighbor identification is substantially more effective than selecting especially close neighbors from a small fixed neighborhood.The alternative moves very close neighbors together while pushing somewhat farther neighbors apart, but produces less useful representations.
- Ablations: DBSCAN produces trivial representations across many parameter settings, limiting practical clustering alternatives in LA's high-dimensional, variable-density embedding space.The method therefore uses k-means for scalable cluster-based identification.
- Ablations: Lower skewness in close-neighbor clusterings is associated with better performance, while optimal close-neighbor size scales with network depth.More clusterings can average out random outliers, but excessive clustering can create clusters too large for the network capacity.
6. Discussion
The discussion presents LA as an objective balancing attraction among similar inputs with separation of dissimilar inputs. Applied to DCNNs, it yields representations useful for transfer learning across challenging visual tasks, while future work targets broader priors, learning dynamics, stochastic embeddings, and non-image domains.
- 6. Discussion: LA balances bringing similar inputs together with allowing dissimilar inputs to move apart.The objective combines ideas from recent unsupervised-learning advances.
- 6. Discussion: Applied to DCNNs, LA creates representations useful for transfer learning across challenging visual tasks.The paper analyzes the procedure to provide intuition for how it works.
- 6. Discussion: Future work includes nonlocal similarity priors, multi-step representational-change measures, nondeterministic embeddings, and applications beyond images.The proposed extensions include video and audio signals.
A. Clustering Combination
The supplementary material illustrates combining multiple clusterings for robustness and reports nearest-neighbor validation results supporting hyperparameter choices across architectures.
- A. Clustering Combination: Nearest-neighbor validation results support the selected H and m settings for different architectures.The supplementary material reports these comparisons in Table 1.
- A. Clustering Combination: Combining multiple clusterings is illustrated as a way to achieve robustness.The target vector is shown with close neighbors under specified hyperparameter settings.
B. Results Details
The supplementary results document additional data-augmentation settings used during transfer learning on ImageNet and Places 205.
- B. Results Details: Transfer learning generally uses random crops and random horizontal flips on ImageNet and Places 205.The specific random-crop implementation varies across networks and datasets.
B.2. DeepCluster Results Details
The DeepCluster comparison used author-produced implementations across ResNet-18 and ResNet-50 variants, with the reported results selected from these implementations. Training and transfer-learning settings largely followed the provided source-code configurations.
- DeepCluster results for VGG16, ResNet-18, and ResNet-50 were produced by the authors using provided networks or source codes.
- Two ResNet-18 implementations removed the final pooling and fully connected layers before adding fully connected layers, including a 10000-unit final layer.
- ResNet-50-B could not be trained because its k-means clustering became trivial at the third epoch, so the paper reports ResNet-50-A results.
- Most network-training and transfer-learning hyper-parameters matched the provided source-code settings, except for data augmentation techniques.
B.3. Places KNN Results
Places 205 representations were evaluated with a KNN validation procedure using a memory bank generated from center crops. Additional experiments examined how adjustable Local Aggregation training hyper-parameters influenced results.
- Places 205 training images were center-cropped to generate the memory bank used for KNN validation.
- The Places 205 KNN validation procedure followed the ImageNet KNN procedure described in the main paper.
- The Faster R-CNN implementations used SGD with momentum 0.9, batch size 256, weight decay 0.0001, and a learning rate reduced tenfold after 50000 steps.
- Experiments in Table 3 examined the effects of clustering-result update frequency, the parameter k in N_k for B_i, and clustering inputs.