Source-linked AI summary
Invariant Information Clustering for Unsupervised Image Classification and Segmentation
Xu Ji, João F. Henriques, Andrea Vedaldi
TL;DR
Unsupervised deep clustering must avoid costly labels and degenerate solutions, particularly in segmentation. IIC trains a classifier directly by maximising mutual information between paired samples, using random transformations or spatial proximity to create pairs. It achieves state-of-the-art results across classification and segmentation benchmarks, including 88.8% STL10 accuracy in a semi-supervised setting and robustness to removing 90% of labels.
Problem
Manual labels are costly, while straightforward combinations of deep representation learning and clustering can produce degenerate solutions requiring cumbersome external pipelines.
Method
IIC directly trains a randomly initialised classifier without labels by maximising mutual information between assignments for paired samples.
Results
IIC sets state-of-the-art results across unsupervised clustering and segmentation, including 59.6% on STL10, 61.7% on CIFAR10 and 72.3% on COCO-Stuff-3.
Takeaways & Limitations
The resulting networks directly classify and segment images semantically without labels or heuristics, while the method is presented as applicable beyond vision.
Takeaways & Limitations
Without a bottleneck, unbounded model capacity makes the mutual-information objective trivially solvable by the identity function.
Abstract
from arXiv · showhide
We present a novel clustering objective that learns a neural network classifier from scratch, given only unlabelled data samples. The model discovers clusters that accurately match semantic classes, achieving state-of-the-art results in eight unsupervised clustering benchmarks spanning image classification and segmentation. These include STL10, an unsupervised variant of ImageNet, and CIFAR10, where we significantly beat the accuracy of our closest competitors by 6.6 and 9.5 absolute percentage points respectively. The method is not specialised to computer vision and operates on any paired dataset samples; in our experiments we use random transforms to obtain a pair from each image. The trained network directly outputs semantic labels, rather than high dimensional representations that need external processing to be usable for semantic clustering. The objective is simply to maximise mutual information between the class assignments of each pair. It is easy to implement and rigorously grounded in information theory, meaning we effortlessly avoid degenerate solutions that other clustering methods are susceptible to. In addition to the fully unsupervised mode, we also test two semi-supervised settings. The first achieves 88.8% accuracy on STL10 classification, setting a new global state-of-the-art over all existing methods (whether supervised, semi-supervised or unsupervised). The second shows robustness to 90% reductions in label coverage, of relevance to applications that wish to make use of small amounts of labels. github.com/xu-ji/IIC
1. Introduction
IIC addresses degenerate unsupervised clustering by directly training a neural network on paired unlabelled samples with mutual information, avoiding external clustering pipelines. Across image classification and segmentation benchmarks, it achieves state-of-the-art semantic clustering, including strong semi-supervised results.
- Motivation: Large manual annotation costs motivate unsupervised clustering, especially for pixelwise segmentation.Segmentation annotations are particularly expensive because they require pixelwise classification.
- Method: IIC directly trains a randomly initialised neural network end-to-end without labels or heuristics.The network learns a classification function rather than requiring pre-training, feature post-processing, or external clustering.
- Method: The objective maximises mutual information between classifications of paired samples, using random transformations or spatially related patches to form pairs.The objective is applicable to paired data beyond vision and supports an efficient convolutional formulation for segmentation.
- Method: IIC avoids clustering degeneracy through mutual-information entropy maximisation and uses auxiliary overclustering to handle noisy or distractor classes.The main output is trained for relevant classes while the auxiliary output uses more clusters and is ignored at test time.
- Results: 59.6%, 61.7% and 72.3% are reported on STL10, CIFAR10 and COCO-Stuff-3, versus closest competitors at 53.0%, 52.2% and 54.0%.The paper reports new state-of-the-art results across unsupervised clustering and segmentation benchmarks.
- Results: 88.8% accuracy on STL10 establishes a reported global state-of-the-art, while semi-supervised accuracy remains robust when 90% of labels are removed.These settings build on networks trained in an entirely unsupervised fashion.
2. Related work
Prior work distinguishes semantic clustering, which directly produces discrete semantic assignments, from representation learning and auxiliary clustering objectives that require post-processing and may suffer degeneracy. IIC instead uses information-based invariance for paired data, while extending this principle to image transformations and segmentation.
- Co-clustering and mutual information: Information-based deep methods differ in whether information is the sole criterion and whether mutual information is computed exactly for discrete variables or estimated for continuous variables.IIC uses discrete cluster assignments, unlike IMSAT and DeepINFOMAX.
- Semantic clustering versus intermediate representation learning: Semantic clustering directly outputs discrete assignments, whereas intermediate representation learners produce high-dimensional features requiring post-processing such as k-means.DeepCluster uses clustering as an auxiliary proxy rather than directly producing semantically meaningful groups.
- Optimising image-to-image distance: The paper applies paired-data clustering to image clustering and segmentation, generating pairs through random transformations and spatial proximity.This connects the generic objective to both whole-image and pixelwise settings.
- Invariance as a training objective: IIC shares transformation-invariance objectives with methods using exemplars, proximity prediction, denoising, temporal slowness, and local image transformations.These methods model persistence under spatio-temporal or non-material distortions.
3. Method
IIC trains a neural network to assign discrete cluster probabilities by maximizing mutual information between paired samples, using bottlenecks and entropy to avoid degenerate assignments. Random transformations provide image pairs, while auxiliary heads and spatially aligned patch pairs extend the method to image clustering and segmentation.
- Invariant Information Clustering: IIC is a generic objective for clustering unlabelled paired data by training a network to predict cluster identities.For images, pairs are generated with random transformations; segmentation additionally uses spatial proximity.
- Invariant Information Clustering: The learned representation preserves information shared by paired samples while discarding instance-specific details.A finite cluster space provides the bottleneck needed for this compression.
- Invariant Information Clustering: The network uses a softmax output whose C probabilities represent a distribution over discrete cluster assignments.Soft clustering permits uncertainty in the assignment of an input.
- Why degenerate solutions are avoided: Mutual information makes paired assignments predictable while entropy encourages balanced use of clusters, preventing an all-samples-one-cluster solution.Its conditional-entropy term also favors deterministic one-hot predictions rather than uniform outputs.
- Image clustering: For image clustering, IIC pairs each image with a randomly perturbed version and learns clusters invariant to transformations likely to preserve content.Examples include scaling, skewing, rotation, flipping, contrast, and saturation changes.
- Auxiliary overclustering: On datasets containing relevant and distractor classes, an auxiliary overclustering head trains on the full dataset while the main head trains on relevant classes.This uses the larger noisy subset without making it the target of the main clusterer.
- Image clustering: The objective is implemented from predicted cluster-probability matrices for sampled image pairs and their transformed versions.The paired predictions are CNN softmax outputs.
- Image segmentation: For segmentation, IIC clusters receptive-field patches and aligns transformed probability tensors so corresponding spatial patches form pairs.The objective maximizes information between each patch label and its transformed neighbour across images, patches, transformations, and neighbour displacements.
4. Experiments
IIC is evaluated on unsupervised and semi-supervised image clustering and segmentation, outperforming baselines while remaining robust when labels are scarce.
- Experiments: IIC is applied to fully unsupervised image clustering and segmentation, plus semi-supervised finetuning and overclustering.The experiments cover STL10, CIFAR10, CIFAR100-20, MNIST, COCO-Stuff, and Potsdam.
- Image clustering: IIC discovers semantic clusters on unlabelled data and outperforms competing baselines, with margins of 6.6% on STL10 and 9.5% on CIFAR10.The evaluation uses a best one-to-one permutation mapping between learned and ground-truth clusters.
- Image clustering: Semi-supervised overclustering remains robust after discarding 90% of STL10 labels, costing approximately 10% in accuracy.The setting reduces available labelled data from 5000 to 500 over 10 classes.
- Segmentation: Without labels or heuristics, IIC partitions COCO-Stuff-3 into sky, vegetation, and ground, and classifies vegetation, roads, and buildings on Potsdam-3.The segmentations are locally coherent and capture fine detail without post-processing; IIC outperforms all baselines by 18.3% on COCO-Stuff-3.
- Segmentation: IIC's convolutional loss processes all pixels in batch images in parallel, while baselines required pixel sampling and trained up to 3.3× slower.The reported speed advantage accompanies dense segmentation experiments.
5. Conclusions
The paper concludes that mutual-information optimization over related pairs can train neural networks into semantic clusterers without labels or heuristics, achieving state-of-the-art image classification and segmentation.
- Conclusions: IIC trains neural networks into semantic clusterers without labels or heuristics by optimizing mutual information between related pairs.Random transforms can generate the related pairs, and the objective naturally avoids degenerate solutions.
- Conclusions: The resulting models classify and segment images with state-of-the-art semantic accuracy.The method is not specific to vision and suggests research directions involving information in datastreams over time.