PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Unsupervised Learning of Deep Representations and Image Clusters

Jianwei Yang, Devi Parikh, Dhruv Batra

arXiv:1604.03628v3cs.CVcs.LG

TL;DR

Unlabeled-image clustering and representation learning are difficult to optimize together despite their mutual relevance. The paper proposes a recurrent CNN–agglomerative-clustering framework with joint forward and backward updates, and reports stronger clustering and transferable representations across datasets and tasks.

  • Problem

    The paper addresses how to jointly learn image clusters and deep representations from unlabeled images, where clustering and representation learning can provide supervisory signals to each other.

  • Method

    The method combines CNN representations with agglomerative clustering in a recurrent framework, updating clusters forward, representations backward, and optimizing them with a unified weighted triplet loss.

  • Results

    The experiments report more precise image clusters and discriminative representations that generalize across datasets and tasks, outperforming previous image-clustering methods.

  • Takeaways & Limitations

    Jointly integrating clustering and representation learning yields representations that transfer to other tasks and image clusters that perform well across varied datasets.

  • Takeaways & Limitations

    The cluster-level loss requires the entire dataset for estimation, making batch-based optimization difficult.

Abstract

from arXiv · show

In this paper, we propose a recurrent framework for Joint Unsupervised LEarning (JULE) of deep representations and image clusters. In our framework, successive operations in a clustering algorithm are expressed as steps in a recurrent process, stacked on top of representations output by a Convolutional Neural Network (CNN). During training, image clusters and representations are updated jointly: image clustering is conducted in the forward pass, while representation learning in the backward pass. Our key idea behind this framework is that good representations are beneficial to image clustering and clustering results provide supervisory signals to representation learning. By integrating two processes into a single model with a unified weighted triplet loss and optimizing it end-to-end, we can obtain not only more powerful representations, but also more precise image clusters. Extensive experiments show that our method outperforms the state-of-the-art on image clustering across a variety of image datasets. Moreover, the learned representations generalize well when transferred to other tasks.

1. Introduction

The paper targets joint image clustering and deep representation learning from unlabeled images, motivated by the cost and scarcity of manual annotation. It casts alternating clustering and CNN representation updates as a recurrent, end-to-end process.

  • Manual labeling is costly, making clustering and unsupervised learning promising approaches for exploiting unlabeled images.
  • The framework jointly learns image clusters and deep representations, using cluster assignments as supervisory signals and representations to improve clustering.
  • Clustering updates occur in the forward pass, while CNN representation learning occurs in the backward pass within alternating iterations.
  • Agglomerative clustering merges initially reliable over-clusters as CNN representations improve, naturally forming a recurrent process.
  • On MNIST, the method progressively reduces 1,762 initial clusters to 17 and finally 10, while producing more discriminative representations.
  • The contributions include a unified loss for seamless joint optimization and reported improvements in image clustering and transferable representations.

2. Related Work

Prior work spans conventional clustering, deep representation learning, and attempts to connect clustering with representation or codebook learning. Existing approaches commonly rely on handcrafted features or supervised data, while iterative connections to clustering had not yet learned deep representations jointly.

  • Clustering methods include hierarchical and partitional approaches, with K-means, EM, spectral clustering, and NMF-based methods among the latter.
  • Deep representations have improved computer vision tasks, but many approaches learn them through supervised training with large labeled datasets.
  • Earlier efforts iteratively connected clustering with codebook or DBN representation learning, but did not jointly learn deep representations and image clusters as this paper proposes.

3. Approach

JULE casts agglomerative image clustering as a recurrent process over CNN representations, jointly updating cluster labels and CNN parameters. Its forward pass merges clusters using affinity and local-structure criteria, while the backward pass optimizes a cluster-aware loss with stochastic triplets.

  • 3.4. A Recurrent Framework: Agglomerative clustering is interpreted as repeated cluster merging, enabling image clustering and representation learning to share one recurrent framework.The method divides the overall process into partially unrolled periods and alternates forward clustering with backward representation updates.
  • 3.4. A Recurrent Framework: The framework represents images with CNN features and organizes predicted labels into clusters whose states evolve across recurrent timesteps.At each timestep, CNN representations and the previous hidden state generate the current cluster labels.
  • 3.5.1 Forward Pass: At each forward step, the method selects a cluster and nearest neighbor by minimizing a loss that combines direct affinity with local neighborhood structure.The second term compares affinity to the nearest neighbor with affinities to other neighboring clusters, favoring locally distinctive merges.
  • 3.5.1 Forward Pass: Unlike conventional agglomerative clustering, the proposed criterion can select a cluster that is close to its nearest neighbor yet relatively far from other neighbors.The toy example contrasts conventional selection of node b with the proposed selection of node e under local-structure consideration.
  • 3.5.2 Backward Pass: The backward pass accumulates losses across periods and approximates the cluster-level objective with a weighted triplet loss suitable for batch-based stochastic gradient descent.The original cluster loss requires the entire dataset, while the approximation searches neighboring samples from other clusters within a training batch.

4. Experiments

Experiments evaluate JULE across diverse image datasets, clustering algorithms, cross-dataset transfers, face verification, and image classification. The learned representations improve clustering broadly and support transfer to related datasets and recognition tasks.

  • Experimental setup: The evaluation covers digit, object, and face datasets, using normalized mutual information (NMI), where larger values indicate more precise clustering.MNIST, USPS, COIL20, COIL100, UMist, FRGCv2.0, CMU-PIE, and YTF are included.
  • Quantitative comparison: Both OURS-SF and OURS-RC outperform previous methods on all datasets with noticeable margin.OURS achieves NMI = 1 on COIL20 and CMU-PIE.
  • Generalization across clustering algorithms: Hand-crafted features such as HOG, LBP, and spatial pyramids perform worse than the learned representations on tested subsets.
  • Generalization across clustering algorithms: The learned representations improve clustering for all evaluated algorithms and reduce performance variance across algorithms.K-means and AC-Link improve substantially over their performance with raw intensities, and some methods surpass AC-GDL.
  • Cross-dataset clustering: Representations learned on one related dataset improve clustering on another, although transferred performance is below training directly on the target dataset.The experiments use COIL20/COIL100 and USPS/MNIST-test pairs.

5. Conclusion

The conclusion presents JULE as a recurrent framework that jointly learns deep representations and image clusters through alternating clustering and representation-learning steps. Experiments demonstrate its performance across clustering, transfer learning, and classification.

  • JULE combines agglomerative clustering with CNNs in a recurrent process for jointly learning image clusters and deep representations.
  • A partially unrolled process merges clusters during the forward pass and learns representations during the backward pass.
  • A single weighted triplet-loss function guides both clustering and representation learning.
  • Experiments cover image clustering, deep representation transfer learning, and image classification.

A.1. Affinity Measure for Clusters

The affinity measure defines similarity between two clusters using directed affinity terms in both directions. The construction is symmetric between the clusters.

  • The cluster affinity A(C_i, C_j) is formed by adding the directed affinities A(C_j → C_i) and A(C_i → C_j).
  • The affinity matrix W supplies the sample-to-sample relationships used to compute affinities between clusters.W_Ci,Cj and W_Cj,Ci are submatrices connecting samples across the two clusters.
  • The resulting affinity satisfies A(C_i, C_j) = A(C_j, C_i).

A.2. Approximated Affinity Measure

The appendix approximates repeated affinity computations in agglomerative clustering to reduce computational cost. Experiments indicate that a range of α values preserves comparable or better performance while requiring less time.

  • Repeatedly recomputing affinities after cluster merges becomes computationally dominant on large-scale datasets.
  • The approximation estimates the merged-cluster affinity from sample-count ratios rather than fully recomputing all terms.
  • The approximation achieves analogous performance while costing much less time than the original affinity computation.
  • Figure 4 compares NMI and AC across acceleration settings, while Figure 5 compares their time costs.
  • For α in [−0.2, 0.1], accelerated agglomerative clustering achieves analogous or better performance than the unaccelerated baseline.The baseline is the original AC-GDL algorithm using image intensities as input.

A.3. Cluster-based to Sample-based Loss

The cluster-based loss is decomposed into sample-based weighted triplet losses across agglomerative clustering steps. The approximation assigns signs and weights to intra- and inter-cluster affinities using current cluster assignments.

  • At each time step, the sample-based objective uses positive pairs from the same cluster and negative pairs from different clusters.The triplet construction treats samples from the same cluster as positive and samples from another cluster as negative.
  • The loss combines affinity contributions across multiple clustering time steps, including pairs whose cluster relationship changes during merging.Terms from different steps can merge when samples move from separate clusters into one cluster.
  • The cluster-based loss is decomposed recursively from high-level clusters to individual samples for arbitrary agglomerative clustering processes.Different clustering processes yield different sample-affinity weights.
  • The implementation assigns weight γ to same-cluster affinities and weight 1 to different-cluster affinities, emphasizing the signs of the affinities over their scales.The current cluster ids determine which pairs receive each weight.
  • For CNN optimization, samples are selected from the same cluster for positive pairs and from other-cluster neighbors for negative pairs, with a triplet-style margin threshold.This sampling is based on the current clustering result.

A.4. Detailed CNN Architectures in our Paper

The paper uses dataset-specific CNN architectures, adjusting layer depth so the top-layer response map is approximately 10×10.

  • CNN architectures vary by dataset, with different numbers of layers chosen to make the top-layer response map approximately 10×10.The architecture table covers the datasets used in the paper.
  • The architectures may include convolutional, batch-normalization, and weighted-triplet-loss layers.The table marks whether each layer is used for each dataset.

A.5. Performance Evaluated by Accuracy

Accuracy-based evaluation shows the proposed method outperforming other clustering algorithms across the evaluated datasets, while learned representations also improve other clustering methods.

  • The proposed method outperforms other methods on all datasets under clustering accuracy (AC), matching the trend observed with NMI.AC is reported as a supplement to the main-paper NMI evaluation.
  • Using the learned representations boosts the accuracy of the other evaluated clustering algorithms.The results support both clustering performance and the usefulness of the learned representations.
  • The learned deep representations generalize across different clustering algorithms.

A.6. Robustness Analysis

The robustness and supplementary analyses examine parameter sensitivity, initialization reliability, hand-crafted features, visualization, and representation quality across datasets. Results show stable clustering in key parameter ranges, stronger learned representations for clustering, and a visualization limitation in within-cluster structure preservation.

  • Robustness Analysis: Clustering performance changes little as the unfolding rate η increases for most datasets and remains stable for Ks <= 50 across all datasets.Performance drops when Ks becomes larger than this range.
  • Robustness Analysis: Initial over-clusters contain mostly same-category nearest neighbors, supporting the reliability of the algorithm’s starting point.The average initial cluster contains about four samples, and purity is measured using category agreement among K nearest neighbors.
  • Feature Comparisons: Learned representations outperform image intensities and hand-crafted feature pipelines in the reported clustering evaluations.The supplementary comparisons include spatial pyramid, HOG, and LBP features with several clustering algorithms.
  • Low-Dimensional Visualization: The method’s low-dimensional embeddings outperform three compared methods on MNIST 1-nearest-neighbor classification error across three embedding dimensions.The reported comparison is against PCA, NCA, and parametric t-SNE.
  • Low-Dimensional Visualization: The method is less effective than parametric t-SNE on trustworthiness because it emphasizes between-cluster local structure while omitting within-cluster relations.The resulting embeddings may discriminate clusters while disordering samples within a cluster.
  • Learned Representations: Across representation stages, learned embeddings become more discriminative and clustering becomes more precise as iterations proceed.The visualizations show first-three-principal-component representations and predicted cluster counts at different stages.
Loading 1604.03628v3…