Source-linked AI summary
Learning to Discover Novel Visual Categories via Deep Transfer Clustering
Kai Han, Andrea Vedaldi, Andrew Zisserman
TL;DR
The paper studies how to discover novel object categories in unlabelled images when related labelled classes are available. It extends deep clustering with transferred knowledge and estimates the unknown number of categories using labelled probes, substantially outperforming state-of-the-art methods across several benchmarks.
Problem
The central problem is discovering and recognizing novel categories in unlabelled data when category identities and the number of categories are unknown.
Method
The method extends Deep Embedded Clustering with knowledge from labelled classes, a representation bottleneck, temporal ensembling, consistency, and probe-based class-number estimation.
Results
The method substantially outperformed state-of-the-art techniques on ImageNet, OmniGlot, CIFAR-100, CIFAR-10, and SVHN in known and unknown category-number settings.
Takeaways & Limitations
Deep transfer clustering can jointly learn representations and clusters for novel unlabelled categories while leveraging related labelled categories.
Takeaways & Limitations
The labelled and unlabelled class sets are assumed to differ in identity and number, with knowledge transferred as general properties of classes rather than specific class identities.
Abstract
from arXiv · showhide
We consider the problem of discovering novel object categories in an image collection. While these images are unlabelled, we also assume prior knowledge of related but different image classes. We use such prior knowledge to reduce the ambiguity of clustering, and improve the quality of the newly discovered classes. Our contributions are twofold. The first contribution is to extend Deep Embedded Clustering to a transfer learning setting; we also improve the algorithm by introducing a representation bottleneck, temporal ensembling, and consistency. The second contribution is a method to estimate the number of classes in the unlabelled data. This also transfers knowledge from the known classes, using them as probes to diagnose different choices for the number of classes in the unlabelled subset. We thoroughly evaluate our method, substantially outperforming state-of-the-art techniques in a large number of benchmarks, including ImageNet, OmniGlot, CIFAR-100, CIFAR-10, and SVHN.
1. Introduction
The paper addresses discovering and recognizing novel visual categories from unlabelled images by transferring knowledge from labelled, related classes. It introduces deep transfer clustering and a probe-based mechanism to estimate the number of novel categories, achieving strong benchmark performance.
- Motivation: Machines struggle to organize image data into object categories without manually labelled examples or predefined category definitions.The paper contrasts strong supervised recognition with limited open-world, unsupervised category discovery.
- Problem setting: The task transfers knowledge from known labelled categories to discover and recognize different novel categories in unlabelled images.The labelled and unlabelled categories differ, so the goal is to learn general properties of classes rather than specific class identities.
- Method: Deep transfer clustering modifies Deep Embedded Clustering to guide representation learning and cluster assignment using known classes.The method adds a representational bottleneck, temporal ensembling, and consistency constraints.
- Method: The class-number estimator uses known classes as probes, evaluates candidate counts using probe accuracy and unlabelled-cluster quality, and selects a reliable estimate.Probe examples are added to the unlabelled set as if unlabelled before clustering candidates are evaluated.
- Evaluation: The approach was evaluated on ImageNet, OmniGlot, CIFAR-100, CIFAR-10, and SVHN, outperforming competing methods by a substantial margin.The reported evaluation covers public benchmarks and supports performance gains across the listed cases.
2. Related work
The work connects novel-category discovery to semi-supervised learning, transfer learning, clustering, metric learning, and domain adaptation. Its setting differs because target images are unlabelled novel categories rather than labelled shared classes or merely shifted domains.
- Semi-supervised learning: Semi-supervised learning assumes labelled and unlabelled data share the same categories, unlike this novel-category discovery setting.The paper notes that this shared-category assumption is invalid when unlabelled data contains new categories.
- Transfer learning: Transfer learning transfers knowledge between datasets with different categories, while this work transfers knowledge to an unlabelled target dataset.The paper identifies the unlabelled target as the key distinction from conventional transfer learning.
- Clustering: Deep Embedded Clustering is a relevant deep-clustering approach because it jointly clusters data and learns its representation.The proposed method modifies DEC so clustering can be guided by known classes.
- Metric learning: Metric learning provides initialization but produces continuous embeddings, whereas this task requires a discrete partition and exploits unlabelled data.The paper states that converting metric-learning embeddings into discrete classes is often nontrivial.
- Domain adaptation: Domain adaptation addresses source-target domain discrepancy while generally assuming shared classes; this work instead uses novel unlabelled categories on the same manifold.The paper characterizes its setting as more practical but more challenging than standard domain adaptation.
- Closest prior work: The closest prior approaches formulate novel visual category discovery as deep transfer clustering, including a two-stage constrained clustering network.The cited constrained clustering network first learns pairwise similarity from labelled data, then clusters unlabelled data.
3. Deep transfer clustering
Deep transfer clustering uses known classes to guide discovery of different, unlabeled classes while learning their representation and estimating their unknown number. It extends DEC with transfer, bottleneck, temporal ensembling, and consistency mechanisms, then uses probe classes and cluster-quality indices for cardinality selection.
- Problem: The task assigns unlabeled images to unknown classes while using labeled classes that differ in identity and number to learn general properties of good classes.The labeled data guide discovery of new classes rather than supplying the target class identities directly.
- Overview: The method has two components: transfer-guided deep clustering and reliable estimation of the number of unlabeled classes K.The first extends a deep clustering algorithm; the second estimates K for the unlabeled data.
- Joint clustering and representation learning: DEC simultaneously learns embeddings and clusters, with the representation initialized on labeled data and fine-tuned on unlabeled data.The embedding is produced by a neural network fθ, while DEC represents clusters with prototype vectors.
- Joint clustering and representation learning: The clustering procedure sharpens assignments by squaring p(k|i) and normalizing by cluster frequency, reinforcing assignments while balancing cluster sizes.It alternates representation optimization with target-distribution updates.
- Transfer and bottleneck: Transfer is implemented by pretraining the representation on labeled data, while PCA reduces features to K dimensions before K-means initializes cluster centers.The PCA layer is retained as the deep network head.
- Temporal ensembling and consistency: Temporal ensembling and consistency add a gradually ramped constraint matching predictions across transformed or temporally ensembled samples.The consistency weight increases from 0 to 1 during training.
- Estimating the number of classes: Class-number estimation clusters unlabeled data with labeled probe classes across candidate K values, then selects K using probe-set accuracy and an unlabeled-data validity index.The final clustering drops outlier clusters containing less than a threshold fraction of the largest clusters.
4. Experimental results
The experiments evaluate Deep Transfer Clustering across multiple benchmarks, known and unknown category-count settings, and transfer from ImageNet features. The method substantially outperforms baselines while estimating unknown category counts effectively, though transfer quality depends on source–target relatedness.
- Evaluation settings: The evaluation covers known-category discovery on OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN, plus unknown-category discovery on OmniGlot, ImageNet, and CIFAR-100.The unknown-category experiments separate a probe set from labelled classes.
- Known category counts: The learned CIFAR-10 representation separates novel classes in t-SNE space, while errors remain between horses and dogs and between trucks and ships.The reported confusions are associated with similar pose, color, or scene context.
- Unknown category counts: With unknown category counts, the estimator has average error below 5 across all three datasets and enables the method to outperform previous state-of-the-art by 6.8% ACC on OmniGlot and 6.1% ACC on ImageNet.The estimated counts are used before running the transfer clustering algorithm.
- Unknown category counts: Using estimated cluster counts improves KCL and MCL significantly in ACC, while the proposed method still significantly outperforms both augmented methods on all metrics.The comparison is reported for OmniGlot and ImageNet.
- Transfer from ImageNet: ImageNet-pretrained transfer consistently improves CIFAR-10 clustering over k-means and the baseline, but performs poorly on SVHN, likely because ImageNet and SVHN have little correlation.DTC-Π performs best in ACC and DTC-TE performs best in NMI on CIFAR-10.
5. Conclusion
The paper frames novel visual category discovery as deep transfer clustering, learning representations and clusters for unlabelled novel categories while using related labelled categories. It also estimates the unknown category count and reports substantial benchmark gains over state-of-the-art methods.
- The approach simultaneously learns a data representation and clusters unlabelled images from novel visual categories.
- It transfers knowledge from related labelled categories to guide novel-category discovery.
- The method estimates the number of unlabelled categories using labelled probe data.
- The method substantially outperformed state-of-the-art techniques when the category number was known or unknown.
A. Bottleneck dimension
The bottleneck dimension is evaluated through ACC and NMI on a 10-class unlabelled CIFAR100 subset. Performance is relatively insensitive to the dimension, with c = K giving the best observed results.
- The experiment varies bottleneck dimension c for the Ours-Baseline model on a 10-class unlabelled CIFAR100 subset.The learned representation is reduced from d to c dimensions.
- ACC and NMI are not very sensitive to c, especially when c is slightly larger than the number of unlabelled categories K.Here, K = 10.
- c = K gives the best ACC and NMI in this experiment.
B. Number of clusters
The clustering model is evaluated with different assumed numbers of clusters on a 10-class unlabelled CIFAR100 subset. The algorithm estimates 12 categories, close to the ground truth of 10, while mismatches reduce ACC and NMI.
- The algorithm estimates 12 categories for the 10-class unlabelled CIFAR100 subset.The estimate is close to the ground truth of 10.
- ACC and NMI decrease when the assumed number of categories differs from the ground truth.
- Larger cluster counts are preferable to smaller ones because ACC and NMI decrease faster with smaller counts.
C. Representation visualization
The learned representation is visualized in two dimensions for five unlabelled CIFAR10 classes. The pretrained feature extractor does not properly separate the novel classes or produce clear class boundaries.
- The data points are colored according to their ground-truth labels.
- The visualization uses t-SNE to project unlabelled dog, frog, horse, ship, and truck images into two dimensions.
- Representations from the feature extractor pretrained on labelled data do not properly distinguish the novel classes.
D. Evolving of soft clustering assignment
The model learns novel categories through a self-evolving soft clustering assignment, constructing target distribution Q from prediction P. Figure 6 tracks this evolution across unlabelled CIFAR-10 instances.
- Figure 6 plots assignment probabilities horizontally for each instance identity on the vertical axis.Instances are grouped by identity into dog, frog, horse, ship, and truck subsets in the CIFAR-10 example.
- Training is driven by a self-evolving soft clustering assignment.The assignment gradually supports discovery of novel categories.
- Target distribution Q is constructed from prediction P to guide clustering during training.P denotes the soft clustering assignment, while Q is the constructed target distribution.
E. t-SNE visualization with images
Figure 7 visualizes the unlabelled CIFAR-10 subset using the model’s learned representation, combining embedding projections with corresponding input images.
- The visualization covers CIFAR-10 images from the dog, frog, horse, ship, and truck categories.
- The upper panel shows an embedding projection colored by ground-truth labels.
- The lower panel associates learned features with their corresponding images.
F. Detailed category number estimation results on OmniGlot
The paper contrasts explicit pre-clustering category-number estimation with methods that infer the number of categories only after clustering. OmniGlot results are presented in a dedicated table.
- KCL and MCL assume 100 categories rather than explicitly estimating the category count beforehand.
- KCL and MCL estimate categories after clustering by counting clusters whose assigned instances exceed a threshold.
- The proposed approach estimates the number of categories before transfer clustering using Algorithm 2 with Kmax = 100.
- Table 8 reports category-number estimation results on OmniGlot.