PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to cluster in order to transfer across domains and tasks

Yen-Chang Hsu, Zhaoyang Lv, Zsolt Kira

arXiv:1711.10125v3cs.LGcs.AIcs.CV

TL;DR

The paper addresses transfer when target labels are unavailable, asking what can be transferred beyond features across domains and tasks. It learns a transferable pairwise similarity function and uses it in a robust clustering network, achieving strong cross-task and cross-domain results, with further gains when combined with domain discrepancy loss.

  • Problem

    The paper asks what knowledge beyond features can support cross-domain and cross-task transfer when target labels are absent.

  • Method

    The method converts categorical information into pairwise similarity, learns a similarity function, and uses a constrained-clustering-inspired objective to jointly optimize features and clustering.

  • Results

    The approach achieves state-of-the-art clustering on Omniglot and ImageNet, improves Office-31 over state-of-the-art methods, and reaches 77.5% average accuracy versus 71.3% for the source-only baseline.

  • Takeaways & Limitations

    Predicted semantic similarity can benefit unsupervised transfer across tasks and domains, even without explicit domain adaptation, while combining a domain discrepancy loss provides further improvement.

  • Takeaways & Limitations

    Performance depends on similarity-prediction quality, and the paper does not explore settings where learning that function is harder.

Abstract

from arXiv · show

This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster. The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning. We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not. This similarity is category-agnostic and can be learned from data in the source domain using a similarity network. We then present two novel approaches for performing transfer learning using this similarity function. First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs. Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network. Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches. Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet. Our results show that we can reconstruct semantic clusters with high accuracy. We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets. Our approach doesn't explicitly deal with domain discrepancy. If we combine with a domain adaptation loss, it shows further improvement.

1 INTRODUCTION

The paper asks what knowledge beyond features can support transfer when target labels are unavailable, and proposes transferable similarity within a learnable clustering objective. This framework supports both cross-task and cross-domain transfer, with jointly optimized features and clustering.

  • The paper focuses on transferring knowledge beyond features when target labels are absent, using a learned similarity function as clustering’s fundamental component.
  • A neural clustering network trained on similarity outputs supports both cross-task and cross-domain transfer.
  • The method formulates clustering with a transferable similarity-prediction term and optimizes features and clustering jointly end-to-end.
  • The approach transfers predictive pairwise similarity and deploys its objective in cross-task and cross-domain scenarios with deep neural networks.
  • Experiments report state-of-the-art clustering on Omniglot and ImageNet, improvements on Office-31 without explicit domain adaptation, and further gains when domain adaptation is added.

2 RELATED WORK

Prior work transfers features, learns metrics from pairwise constraints, or minimizes domain discrepancy, while this paper transfers predicted relational knowledge for jointly learned constrained clustering. Its comparison places the method among a third group that directly incorporates constraints into clustering.

  • Transfer learning commonly moves instances, features, model parameters, or relational knowledge; this paper proposes pairwise similarity as transferred meta-knowledge between parameters and relational information.
  • Cross-task work has mainly transferred ImageNet-trained features to supervised vision tasks, whereas this paper investigates transfer across tasks without relying only on features.
  • Domain adaptation methods typically address domain shift by minimizing feature-distribution discrepancy, a strategy this work complements by transferring extra information.
  • Constrained-clustering methods include metric-learning approaches requiring a separate clustering stage and methods that formulate the clustering objective directly from constraints.
  • The proposed strategy belongs to a third constrained-clustering group and evaluates it against four groups using densely predicted rather than sparsely available ground-truth constraints.

3 THE TRANSFER LEARNING TASKS

The paper defines unsupervised transfer with labeled source information and an unlabeled target, distinguishing cross-task category changes from cross-domain distribution shifts. It transforms cross-task categorization into pairwise similarity prediction and uses that information to reconstruct target clusters, while cross-domain transfer combines auxiliary, source, and target data.

  • Unsupervised transfer uses labeled source data S and unlabeled target data T, with cross-task transfer changing categories and cross-domain transfer retaining categories under shifted input distributions.
  • The source is split into S′ and auxiliary dataset A; cross-task transfer uses A and unlabeled T, whereas cross-domain transfer uses A, S′, and T.
  • When target categories differ, the method reduces categorization to a surrogate same-task problem and reconstructs target cluster structure from transformed-task predictions.
  • The auxiliary labels are converted into similar or dissimilar pairs, which train a category-agnostic similarity function G(x_i, x_j) = y_i,j.
  • Figure 2 presents cross-task transfer as a counter-clockwise conversion to cross-domain transfer, with G and constrained clustering as its key components.
  • Applying G to target pairs produces pairwise predictions used to infer target categories, whose cluster indices may be arbitrarily ordered but should contain coherent semantic categories.
  • For cross-domain transfer, ImageNet serves as labeled auxiliary data, one Office-31 domain is labeled source data, and another Office-31 domain is unlabeled target data.

4 THE LEARNABLE CLUSTERING OBJECTIVE (LCO)

The learnable clustering objective uses predicted pairwise similarity to train a robust clustering network for cross-task and cross-domain transfer. Its losses jointly optimize feature representations and cluster assignments, with optional classification and domain-adaptation components.

  • Objective design: Pairwise similarity is binarized into similar and dissimilar constraints and used to construct a contrastive clustering objective.The similarity predictions may be noisy, so the objective is designed for constrained clustering with pairwise information.
  • Objective design: The objective compares cluster-assignment distributions with pairwise KL-divergence, encouraging similar instances to agree and dissimilar instances to differ.For each pair, the neural network produces distributions P = f(xp) and Q = f(xq); dissimilar pairs use a hinge-loss formulation.
  • Objective design: LCO requires neither cluster centers nor a predefined feature metric, allowing feature representation and clustering to be jointly optimized by back-propagation.The divergence is computed directly on cluster assignments rather than on the feature representation.
  • Similarity prediction: The similarity network G predicts image-level semantic similarity from category-derived binary supervision and outputs pair predictions for all similarity pairs in a mini-batch.G is trained on an auxiliary dataset and its output probability g ranges from 0 to 1, with 1 indicating greater similarity.
  • Transfer settings: The same clustering objective supports cross-task learning with unlabeled target data and cross-domain transfer by combining target clustering with source classification loss.For cross-domain transfer, source and target images are mixed in a mini-batch; the approach transfers constraints without explicitly matching source and target feature distributions.

5 EXPERIMENTS

Experiments evaluate the method on unsupervised cross-task and cross-domain transfer, emphasizing reconstruction with noisy similarities and robustness when the cluster count is unknown. The approach achieves strong clustering and domain-adaptation performance across Omniglot and Office-31.

  • Experiments cover four image datasets and evaluate both cross-task and cross-domain transfer schemes.
  • Cross-task transfer: The Constrained Clustering Network reconstructs target semantic clusters from similarity outputs, with an output layer sized for potential target clusters.
  • Cross-task transfer: 78.1% average accuracy is achieved when K is unknown, compared with 82.4% when K is known and 65.4% for CSP.
  • Cross-task transfer: CCN remains robust to noisy predicted similarities, while jointly optimizing metric learning and clustering outperforms using constraints only for metric learning.
  • Cross-task transfer: 6.35 average difference is achieved for estimating the number of clusters, compared with 16.3 for SKMS; 10 of 20 datasets have differences of at most 3.
  • Cross-domain transfer: 77.5% average accuracy is achieved on Office-31, a 6.2-point gain over the 71.3% source-only baseline and above DANN at 75.7% and JAN at 76.9%.

6 CONCLUSION AND OUTLOOK

The paper transfers pairwise similarity predictions for cross-task and cross-domain learning, with benefits increasing when domain discrepancy loss is added. Its performance depends on constrained-clustering robustness and similarity prediction quality, with harder settings left for future work.

  • Pairwise similarity predictions transfer knowledge across cross-task and cross-domain learning settings.The method uses a constrained-clustering-inspired loss within a neural network that jointly optimizes features and clustering outputs.
  • The approach achieves strong results across several datasets without explicit domain adaptation, with further benefits from adding domain discrepancy loss.
  • Performance depends on constrained-clustering robustness and the quality of the similarity prediction function.
  • The paper does not explore cases where learning similarity is harder, such as few source categories or large source–target domain discrepancy.The authors identify domain-adaptive learning of G and related settings as future work.

A COMPARISON OF EXPERIMENTAL SETTINGS

The experiments vary datasets, transferred similarity functions, clustering-network optimization data, losses, and network initialization across cross-task and cross-domain settings. The figures illustrate cross-task transfer with exclusive classes and domain adaptation on Office-31.

  • Table 3 specifies dataset roles, including dataset A for learning G and datasets T or T∪S′ for optimizing CCN variants.CCN* denotes cross-task transfer, while CCN+/++ denotes cross-domain transfer.
  • The loss settings combine the learnable clustering objective with cross-entropy and, for domain adaptation, a domain adaptation loss.G is trained with two-class cross-entropy for similar versus dissimilar pairs.
  • The cross-task experiments transfer between Omniglotbg and Omnigloteval, and between ImageNet882 and ImageNet118, whose classes are exclusive.The ImageNet backbone is initialized with ImageNet882-pretrained weights.
  • The domain adaptation comparison uses Office-31 to contrast transferring semantic similarity with minimizing domain discrepancy.

B SUPPLEMENTARY FOR UNSUPERVISED CROSS TASK TRANSFER LEARNING

The supplementary experiments evaluate clustering accuracy, cluster-number estimation, similarity prediction, and unsupervised ImageNet transfer under specified class-count and sampling settings.

  • Clustering accuracy on Omnigloteval is reported by alphabet with K = 100 to simulate an unknown number of clusters.
  • Cluster-number estimates are reported across 20 Omnigloteval datasets, with bold values indicating errors smaller than or equal to 3.ADif is defined in section 5.1.2.
  • The similarity function trained on Omniglotbg is evaluated on Omnigloteval and MNIST using an N-way one-shot-learning test.Raw similarity probabilities are used to select the nearest exemplar.
  • ImageNet cross-task transfer averages results over three random ImageNet118 subsets, each containing 30 classes, with ACC evaluated at K = 30 and ACC (100) at K = 100.All methods use average-pooling features from an ImageNet882-pretrained ResNet-18.
  • Similarity prediction on three ImageNet118 subsets binarizes predictions at 0.5 to calculate precision and recall.The evaluation samples 12M pairs and includes a uniformly random similarity baseline.

C.1 MORE DISCUSSION FOR THE OFFICE-31 EXPERIMENTS

The Office-31 discussion compares backbone depth, similarity prediction across domains, and transfer without pretrained backbone weights. Deeper pretrained networks provide a substantial performance advantage.

  • 8.4 points separate ResNet-50 from AlexNet, exceeding the 6.2 points gained from JAN domain adaptation.The comparison comes from experiment table 10.
  • Memory limitations prevented applying LCO to deeper models, leaving multi-GPU implementations as future work.
  • Office-31 results compare AlexNet with ResNets ranging from 18 to 50 convolutional layers, and deeper features generalize better across domains.Source-only training directly applies the classifier trained on S′ to T.
  • The similarity function G trained on ImageNet882 is evaluated across three Office-31 domains using 1.4M image pairs.
  • The SVHN-to-MNIST and Omniglotbg-to-MNIST settings evaluate unsupervised domain transfer without pretrained backbone weights.The setup is compared with prior source-only configurations.

D ROBUSTNESS ANALYSIS OF CONSTRAINED CLUSTERING NETWORK

The robustness study varies similarity-prediction quality through pair-label recall while evaluating clustering on MNIST with controlled training settings. It uses repeated random restarts to report the best result for each factor combination.

  • MNIST clustering uses 60k training images of size 28x28, with normalized raw pixels fed directly into a small convolutional network.
  • The clustering network is randomly initialized and trained five times for each factor combination, with the best final result reported.Training uses mini-batches of 256 for 15 epochs, with up to 65,536 pairs per full-density mini-batch.
  • Pair-label noise is simulated by flipping ground-truth pair labels according to designated similar- and dissimilar-pair recalls.Recall values for both pair types are gradually reduced from one to zero in intervals of 0.1.

D.2 DISCUSSION

Clustering remains robust to substantial noise in similar-pair predictions, but depends more sharply on dissimilar-pair recall. Higher pairwise density improves robustness, and overestimating the number of clusters causes only slight degradation in MNIST.

  • NMI remains robust below 0.5 similar-pair recall when dissimilar-pair recall is high, whereas performance can completely fail below 0.5 dissimilar-pair recall.When dissimilar-pair recall exceeds 0.5, clustering tolerates substantial noise in dissimilar pairs; similar-pair recall is less critical in that regime.
  • Higher pairwise constraint density substantially enlarges the region of robust clustering performance.At lower density, fewer pairwise relationships contribute to each update, and Figure 8 shows the bright region shrinking significantly.
  • Higher density can let true similar-pair gradients override false similar-pair gradients within a mini-batch, reducing error effects and speeding convergence.
  • Figure 8 encodes NMI with bright colors near 1 and black near 0, while rows vary pairwise density and output cluster count.Each sub-figure shows score changes with similar- and dissimilar-pair recall.
  • Full pairwise enumeration adds negligible GPU computation overhead, with the main cost limited to memory consumption for predicted-distribution vectors.
  • Increasing the softmax output from 10 to 100 clusters only slightly degrades NMI when the true MNIST category count is 10.
Loading 1711.10125v3…