PaperlayerView exact revision on arXiv

Source-linked AI summary

Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation

Yaling Tao, Kentaro Takagi, Kouta Nakata

arXiv:2106.00131v1cs.LGcs.CV

TL;DR

Deep clustering still faces a representation-learning challenge, particularly on complex image data. The paper combines instance discrimination and feature decorrelation, motivated by spectral clustering, and reports 81.5% accuracy on CIFAR-10 and 95.4% on ImageNet-10.

  • Problem

    Representation learning remains a key challenge for deep clustering, and existing methods have fallen short on complex targets such as CIFAR-10.

    Source: §1 · p. 1

  • Method

    The method combines instance discrimination, which learns similarities among instances, with softmax-formulated feature decorrelation, motivated by spectral-clustering properties.

    Source: §1 · p. 2

  • Results

    IDFD, ID(tuned), and IDFO outperform the compared methods except SCAN across datasets on ACC, NMI, and ARI; IDFD reports 81.5% CIFAR-10 accuracy and 95.4% ImageNet-10 accuracy.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    The learned representation achieves accuracies comparable to state-of-the-art values on CIFAR-10 and ImageNet-10 with simple k-means.

    Source: §5 · p. 9

Abstract

from arXiv · show

Clustering is one of the most fundamental tasks in machine learning. Recently, deep clustering has become a major trend in clustering techniques. Representation learning often plays an important role in the effectiveness of deep clustering, and thus can be a principal cause of performance degradation. In this paper, we propose a clustering-friendly representation learning method using instance discrimination and feature decorrelation. Our deep-learning-based representation learning method is motivated by the properties of classical spectral clustering. Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features. We utilize an instance discrimination method in which learning individual instance classes leads to learning similarity among instances. Through detailed experiments and examination, we show that the approach can be adapted to learning a latent space for clustering. We design novel softmax-formulated decorrelation constraints for learning. In evaluations of image clustering using CIFAR-10 and ImageNet-10, our method achieves accuracy of 81.5% and 95.4%, respectively. We also show that the softmax-formulated constraints are compatible with various neural networks.

1 INTRODUCTION

The paper adapts representation learning for complex image clustering by combining instance discrimination and feature decorrelation, guided by spectral clustering. It reports accuracies of 81.5% on CIFAR-10 and 95.4% on ImageNet-10.

1 INTRODUCTION

  • Prior deep-clustering methods had fallen short on complex targets such as CIFAR-10, making representation learning a key challenge.Earlier metric-learning approaches used similarities to assign pseudo-labels or pseudo-graphs, but the authors describe them as heuristic.Source: §1 · p. 1
  • Treating each unlabeled instance as its own class learns discriminative representations that retain similarities among instances.The paper says its softmax formulation works like a spectral-clustering similarity matrix when temperature τ is set higher.Source: §1 · p. 2
  • The softmax-formulated decorrelation constraint is softer than the soft orthogonal constraint and yields stable clustering-performance improvement.The constraints aim to make latent features orthogonal and reduce redundant information.Source: §1 · p. 2
  • 81.5% accuracy on CIFAR-10 and 95.4% on ImageNet-10 are reported for the combined representation-learning method.The approach combines instance discrimination and feature decorrelation.Source: §1 · p. 2
  • The method combines instance-similarity learning with feature decorrelation, drawing on spectral-clustering properties.The paper associates similarity learning with the Laplacian and feature orthogonality with spectral clustering's orthogonality constraints.Source: §1 · p. 2

2 RELATED WORK

Related work spans deep clustering, learned representations, and spectral methods. This paper focuses on learning a representation space suited to clustering rather than directly learning labels or concentrating on the clustering phase.

2 RELATED WORK

  • Deep-clustering approaches range from two-stage representation learning followed by clustering to jointly learned clustering and representation methods.The related work also includes generative-model approaches and reports strong results especially on simple datasets.Source: §2 · p. 2
  • Representation-learning approaches include pairwise-relationship objectives, k-means pseudo-labels, generative models, mutual information, and instance discrimination.Wu et al. (2018) found that discriminating individual instance classes retains similarities among data.Source: §2 · p. 3
  • IIC learns semantic labels directly, while SCAN focuses on the clustering phase using a pre-designed representation; this paper instead targets a clustering-friendly representation space.Source: §2 · p. 3
  • Spectral clustering embeds data in the Laplacian eigenspace derived from pairwise similarities, then applies k-means to the embedding.Shaham et al. incorporated this idea into a neural network using similarities learned from positive and negative pairs.Source: §2 · p. 3

3 PROPOSED METHOD

The method learns clustering-oriented representations by combining instance discrimination with feature decorrelation, drawing on the representation and orthogonality ideas of spectral clustering. A deep network maps inputs to an embedding, which is then clustered in a separate phase.

3.1 INSTANCE DISCRIMINATION

  • Treating each unlabeled instance as its own class trains representations to distinguish instances while retaining similarity useful for clustering.The temperature parameter τ has a large impact on clustering performance.Source: §3.1 · p. 3

3.2 FEATURE DECORRELATION

  • The softmax-formulated constraint encourages feature decorrelation while allowing nonzero off-diagonal correlations, unlike a strict orthogonality constraint.The authors argue that strict orthogonality may be inappropriate when the representation dimension exceeds the unknown underlying structure.Source: §3.2 · p. 4

3.3 OBJECTIVE FUNCTION AND LEARNING MODEL

  • A deep network learns an embedding by combining instance discrimination and feature decorrelation, after which a clustering method such as k-means is applied.The task is divided into representation learning and clustering.Source: §3 · p. 3, §3.3 · p. 5
  • The proposed objective combines instance-discrimination and feature-decorrelation losses, with α balancing their contributions.The paper also describes a variant that combines instance discrimination with a soft orthogonal constraint.Source: §3.3 · p. 5

3.4 CONNECTION WITH SPECTRAL CLUSTERING

  • Feature decorrelation approximates the orthogonality condition in spectral clustering, under which minimizing the instance-discrimination loss approximates minimizing the spectral-clustering loss.The authors describe the combined method as analogous to spectral clustering.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate IDFD’s clustering results, temperature behavior, learned representations, and network compatibility. They report strong benchmark accuracy and describe how the method’s settings and feature constraints affect learning.

4 EXPERIMENTS

  • Experiments use five image datasets and ResNet18 for the main evaluations, with τ = 1 and τ2 = 2 for IDFD.The datasets are CIFAR-10, CIFAR-100, STL-10, ImageNet-10, and ImageNet-Dog.Source: §4 · p. 6

4.1 MAIN RESULTS

  • IDFD achieves higher performance than ID(tuned) throughout learning and is more stable than IDFO, whose ACC fluctuates widely and drops in later epochs.IDFO reaches the highest ACC in earlier epochs.Source: §4.1 · pp. 6–7
  • 81.5% ACC on CIFAR-10 and 95.4% ACC on ImageNet-10; IDFD, IDFO, and ID(tuned) outperform listed methods except SCAN across datasets and metrics.Feature orthogonalization and decorrelation improve performance across all datasets; ID(tuned) also significantly outperforms ID(original) on all datasets.Source: §4.1 · p. 6, Table 1 · p. 7

4.2.1 ANALYSIS ON TEMPERATURE PARAMETER

  • Large τ encourages compact latent distributions, while small τ encourages uniform distributions; experiments show ID accuracy degrades when τ is too small or too large.With τ fixed at 1, IDFD is relatively robust to τ2 and supports stable representation learning.Source: §4.2.1 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8

4.2.2 REPRESENTATION DISTRIBUTION AND FEATURE BEHAVIOR

  • On CIFAR-10, IDFD maintains tighter within-class connections and clearer between-class borders than IDFO, which tends toward patch-like within-class distributions.On ImageNet-10, IDFD representations are described as clustering-friendly, with samples near cluster borders similar in appearance.Source: §4.2.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
  • IDFO decorrelates features, while IDFD retains moderate feature correlation; the softmax constraint alleviates strict orthogonality and enables stable representation learning.The feature correlation comparison is on CIFAR-10 with ResNet18.Source: §4.2.2 · p. 8, Figure 7 · p. 9

4.2.3 INVESTIGATION FOR PRACTICAL USE

  • IDFD works across ConvNet, VGG16, and ResNet34, outperforming ID(tuned); feature decorrelation has a more obvious effect on these networks.The experiments use CIFAR-10; AE does not benefit from the feature constraint as instance discrimination does.Source: §4.2.3 · pp. 8–9, Table 2 · p. 9

5 CONCLUSION

The conclusion presents IDFD as a clustering-friendly representation-learning method combining instance discrimination and feature decorrelation. It reports competitive clustering accuracy and use across neural-network structures.

5 CONCLUSION

  • IDFD combines instance discrimination and feature decorrelation, motivated by spectral-clustering properties, to learn clustering-friendly representations.Instance discrimination learns similarities among data, while feature decorrelation removes redundant correlations among features.Source: §5 · p. 9
  • On CIFAR-10 and ImageNet-10, IDFD with simple k-means achieves accuracy comparable to state-of-the-art values.The conclusion reports that the IDFD loss works across multiple neural-network structures.Source: §5 · p. 9
  • The authors expect IDFD to be effective for various kinds of problems.They designed a softmax-formulated feature-decorrelation constraint to support stable improvement in clustering performance.Source: §5 · p. 9

A DATASETS AND EXPERIMENTAL SETUP

The experiments use five image datasets, with ResNet18 as the main architecture. The setup specifies input resizing, latent dimension, training schedule, loss weights, and temperature settings.

A DATASETS AND EXPERIMENTAL SETUP

  • The experiments cover CIFAR-10, CIFAR-100, STL-10, ImageNet-10, and ImageNet-Dog; STL-10 training and test sets are jointly used.The three ImageNet subsets are resized to 96 × 96 × 3.Source: A DATASETS AND EXPERIMENTAL SETUP · p. 12, Table 3 · p. 12
  • Main experiments use ResNet18 and 128-dimensional latent vectors, trained for 2,000 epochs with batch size 128 and momentum 0.9.The learning rate starts at 0.03 and is scaled down after 600 epochs by 0.1 every 350 epochs.Source: A DATASETS AND EXPERIMENTAL SETUP · p. 12
  • IDFD uses α = 1, τ = 1, and τ2 = 2; IDFO uses dataset-dependent α values, while ID(original) and ID(tuned) use τ = 0.07 and τ = 1.The same ResNet architecture is used across datasets except for the input layer.Source: A DATASETS AND EXPERIMENTAL SETUP · p. 12

B OPTIMAL SOLUTIONS OF CLUSTERING AND INSTANCE DISCRIMINATION

The analysis compares uniform representations spread around a circle with compact representations that group same-class samples. For large τ, the instance-discrimination loss for the uniform case approximates that of the compact case, making instance-level discrimination a reasonable approach for clustering.

B OPTIMAL SOLUTIONS OF CLUSTERING AND INSTANCE DISCRIMINATION

C EXTENDED EXPERIMENTS

This section supplements earlier investigations of the method's practical use with details about several experiments.

C EXTENDED EXPERIMENTS

  • The section adds details about several experiments previously introduced as investigations of practical use.The passage does not specify the experiments or report their findings.Source: C EXTENDED EXPERIMENTS · p. 13

C.1 IMPACT OF NETWORK ARCHITECTURE

IDFD is reported to work across network architectures, with more pronounced gains over ID(tuned) on ConvNet and VGG16 than on ResNet. On CIFAR-10, feature decorrelation worsens autoencoder clustering, unlike its improvement when combined with instance discrimination.

C.1 IMPACT OF NETWORK ARCHITECTURE

  • IDFD outperforms ID(tuned) more substantially on ConvNet and VGG16 than on ResNet, and VGG16 with IDFD has sparse feature correlations.The authors report denser, stronger correlations for VGG16 with ID and attribute the larger feature-decorrelation effect on VGG16 to redundant features that may degrade clustering.Source: §C.1 · p. 13, Figure 10 · p. 14
  • The proposed losses work across network architectures, and the authors expect they can be introduced to networks suited to individual problems.The passage states this as an expectation for practical use.Source: §C.1 · p. 14

C.2 COMBINATION OF AUTOENCODER AND FEATURE DECORRELATION

  • On CIFAR-10, AE scored 26.0% ACC and AEFD scored 22.4% ACC, while ID improved from 26.8% to 42.0% ACC with feature decorrelation.The authors conclude that autoencoder representations do not benefit from feature decorrelation as instance-discrimination representations do.Source: §C.2 · p. 14

C.3 IMPACT OF DATA AUGMENTATION

Data augmentation has a strong impact on clustering performance in the reported experiments. On CIFAR-10, RandomResizedCrop, RandomGrayscale, and ColorJitter have strong effects; the method also performed well on private manufacturing images using generic augmentation.

C.3 IMPACT OF DATA AUGMENTATION

  • Without data augmentation, clustering performance degraded across experiments on five datasets, making augmentation important for practical use.Table 4 reports clustering results with and without augmentation on five datasets.Source: §C.3 · p. 14, Table 4 · p. 14
  • On CIFAR-10, RandomResizedCrop, RandomGrayscale, and ColorJitter have strong effects on image clustering.Figure 11 presents the effect of each data-augmentation technique used for DA on CIFAR-10.Source: §C.3 · p. 14, Figure 11 · p. 15
  • IDFD performed well on private manufacturing images using generic data augmentation, while other data types require corresponding augmentation techniques.The authors name text and time series as examples of other data types.Source: §C.3 · p. 15

Related papers

Loading 2106.00131v1…