PaperlayerView exact revision on arXiv

Source-linked AI summary

Neighborhood Contrastive Learning for Novel Class Discovery

Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, Nicu Sebe

arXiv:2106.10731v1cs.CVcs.AIcs.LG

TL;DR

Novel Class Discovery asks how to cluster unlabeled samples from classes disjoint from a labeled dataset. The paper introduces NCL, which uses neighborhood pseudo-positives and feature-mixed hard negatives in contrastive learning, and reports large gains over state-of-the-art NCD methods, including 13% on CIFAR-100 and 8% on ImageNet.

  • Problem

    Novel Class Discovery seeks to cluster an unlabeled dataset with disjoint classes by leveraging knowledge from a labeled dataset.

  • Method

    NCL learns representations by contrasting queries with augmented views and neighborhood pseudo-positives, while HNG creates hard negatives by mixing labeled and unlabeled features.

  • Results

    Clustering accuracy improves by 13% on CIFAR-100 and 8% on ImageNet over state-of-the-art approaches.

  • Takeaways & Limitations

    Combining neighborhood aggregation with contrastive learning and informative hard negatives significantly improves NCD clustering performance.

  • Takeaways & Limitations

    The pseudo-positive formulation assumes that neighboring examples in ρk are false-negatives belonging to the same class as zu.

Abstract

from arXiv · show

In this paper, we address Novel Class Discovery (NCD), the task of unveiling new classes in a set of unlabeled samples given a labeled dataset with known classes. We exploit the peculiarities of NCD to build a new framework, named Neighborhood Contrastive Learning (NCL), to learn discriminative representations that are important to clustering performance. Our contribution is twofold. First, we find that a feature extractor trained on the labeled set generates representations in which a generic query sample and its neighbors are likely to share the same class. We exploit this observation to retrieve and aggregate pseudo-positive pairs with contrastive learning, thus encouraging the model to learn more discriminative representations. Second, we notice that most of the instances are easily discriminated by the network, contributing less to the contrastive loss. To overcome this issue, we propose to generate hard negatives by mixing labeled and unlabeled samples in the feature space. We experimentally demonstrate that these two ingredients significantly contribute to clustering performance and lead our model to outperform state-of-the-art methods by a large margin (e.g., clustering accuracy +13% on CIFAR-100 and +8% on ImageNet).

1. Introduction

Novel Class Discovery uses labeled data from known classes to cluster an unlabeled dataset with disjoint novel classes. NCL learns discriminative representations by exploiting neighborhood pseudo-positives and feature-mixed hard negatives, achieving large benchmark gains.

  • Motivation: Novel Class Discovery clusters unlabeled samples into novel classes by leveraging a labeled dataset whose class set is disjoint.The labeled data provides latent commonalities, but transferability from supervised features is not guaranteed.
  • Approach: NCL applies contrastive learning to labeled and unlabeled data within one holistic framework for discriminative feature learning.This framework is described as absent from most prior NCD methods.
  • Approach: NCL treats local-neighborhood samples as pseudo-positives and brings them closer to a query, providing more positives than the traditional two-view formulation.The neighborhood assumption is specific to NCD because a feature extractor can first be pretrained with supervision.
  • Approach: Hard Negative Generation mixes labeled samples in feature space to produce harder negatives that aid contrastive learning.The method leverages labeled samples from various classes and reports consistent improvement.
  • Results: Clustering accuracy improves by 13% on CIFAR-100 and 8% on ImageNet over state-of-the-art approaches.The gains are reported from extensive experiments on three NCD benchmarks.

2. Related Work

Related work spans novel class discovery, unsupervised clustering, and contrastive representation learning. NCL combines labeled-and-unlabeled contrastive learning with k-nearest-neighbor pseudo-positives and feature-based hard-negative generation.

  • Novel Class Discovery: Novel Class Discovery clusters unlabeled samples into semantic categories while using labeled data with completely different classes.Prior approaches either initialize clustering with labeled-data training or continue leveraging labeled data during clustering.
  • Unsupervised Clustering: Unsupervised clustering partitions unlabeled data without prior knowledge from a labeled set, using pairwise pseudo-labeling or neighborhood aggregation.NCL draws inspiration from both families of methods.
  • Contrastive Learning: Contrastive learning computes similarity between an input and its correlated view rather than a fixed target such as a one-hot label.NCL adapts this representation-learning formulation to NCD.
  • Novelty: NCL uniquely combines labeled and unlabeled contrastive learning with k-nearest-neighbor pseudo-positives in an end-to-end NCD formulation.This differs from methods that use neighborhood search sensitive to initialization and computationally expensive iterative optimization.
  • Negative Mining: Hard-negative mining supports contrastive learning by using latent-space mixing to generate more difficult negatives.The approach builds on negative-mining strategies that otherwise rely on large batches or queues for useful negatives.

3. Method

The method combines supervised and unlabeled learning in a shared NCD framework, using neighborhood-based pseudo-positives and feature-mixed hard negatives to improve contrastive representations.

  • Problem setup: NCD clusters unlabeled data Du using knowledge from labeled data Dl, whose classes are disjoint but require some similarity.
  • Overall framework: The framework learns a shared feature extractor and two classifiers while training on labeled and unlabeled batches with augmented views.Labeled features use cross-entropy supervision, while unlabeled examples receive pairwise and consistency-based training.
  • Neighborhood Contrastive Learning: NCL retrieves top-k similar features from an unlabeled memory queue and treats them as pseudo-positives for the query.This addresses the risk that same-class samples are treated as negatives in ordinary contrastive learning.
  • Neighborhood Contrastive Learning: Labeled samples use ground-truth classes to retrieve positives, reducing the neighborhood contrastive loss to supervised contrastive learning.
  • Hard Negative Generation: HNG identifies easy true negatives, mixes unlabeled and labeled features, filters the hardest mixed negatives, and adds them to the query-specific queue.The resulting negatives are intended to remain true negatives while being difficult for the network to distinguish from the query.

4. Experiments

Experiments evaluate the datasets, baseline components, NCL and HNG variants, and comparisons with state-of-the-art NCD methods. The full method achieves the strongest reported clustering accuracy across CIFAR-10, CIFAR-100, and ImageNet.

  • Experimental setup: Experiments use CIFAR-10, CIFAR-100, and ImageNet, with results averaged over 10 runs for the CIFAR datasets and 3 runs for ImageNet.
  • Baseline ablation: The baseline ablation finds that removing any component reduces performance, with BCE identified as the most important component.Removing BCE substantially decreases results, while SSL and CS also contribute to representation generality and robustness, respectively.
  • NCL evaluation: NCL gains +5.5% on CIFAR-10 and +12.9% on CIFAR-100, demonstrating the effectiveness of neighborhood-based pseudo-positives.Applying NCL to labeled data and retaining augmented positives further supports discrimination and neighborhood mining.
  • HNG evaluation: HNG significantly increases ACC for CIFAR-100 and improves ACC for ImageNet, while reducing result variance across datasets.HNG provides no boost on CIFAR-10, where the labeled set contains fewer classes and produces less diverse hard negatives.
  • HNG evaluation: HNG generates more varied hard negatives than its variants, whereas mixing only easy unlabeled samples or nearest labeled features is suboptimal.The variant analysis supports using randomly selected labeled features to increase the variety of generated hard features.
  • State-of-the-art comparison: ACC reaches 93.4% on CIFAR-10, 86.6% on CIFAR-100, and 90.7% on ImageNet, exceeding RS by +3%, +13.4%, and +8.2%, respectively.The full method produces new state-of-the-art results for novel class discovery.
  • Visualization: t-SNE visualization shows the initial model has ACC=58.1%, while DTC, RS, and the proposed method produce more discriminative representations with ACC>88%.The proposed method mostly places same-class samples in circular regions and includes indistinguishable samples near class decision boundaries.

5. Conclusion

The paper concludes that NCL learns discriminative features from labeled and unlabeled data by exploiting local neighborhoods. HNG adds informative hard negatives, and experiments show superiority over state-of-the-art NCD methods.

  • The proposed framework applies contrastive learning to labeled and unlabeled data for Novel Class Discovery.
  • NCL leverages local embedding neighborhoods to incorporate more positive samples and improve clustering accuracy.
  • HNG uses labeled samples to produce informative hard negatives that provide further advantage to NCL.
  • Experiments on three datasets demonstrate significant superiority over state-of-the-art NCD methods.

A. Dataset

The experiments use CIFAR-10, CIFAR-100, and ImageNet under NCD splits that separate labeled known classes from unlabeled novel classes. Dataset scale and class partitions vary substantially across benchmarks.

  • CIFAR-10 contains 50,000 training images from 10 classes, with five classes labeled and five classes unlabeled.The labeled classes are airplane, automobile, bird, cat, and deer.
  • ImageNet contains 1.28 million training images from 1,000 classes, split into 882 labeled classes and 118 classes for unlabeled subsets.Three unlabeled subsets are sampled, each containing about 30,000 images from 30 classes.

B.1. Parameter Analysis of NCL

NCL’s performance depends on memory size, temperature, pseudo-positive count, and augmented-positive weight. The analysis selects settings that balance accuracy, robustness, and computational efficiency.

  • Memory size: A memory size between 2,000 and 10,000 yields the best ACC, while 2,000 offers good performance with lower computational cost.The default memory size is set to |M|=2,000.
  • Temperature: NCL is robust to temperatures between 0.02 and 0.1, with the best results around τ=0.05.The default temperature is τ=0.05.
  • Number of pseudo-positives: Too few pseudo-positives misclassify positives as negatives, whereas too many include negatives and reduce ACC.The tested k1 ranges are [10, 360] for CIFAR-10 and [1, 80] for CIFAR-100.
  • Augmented-positive weight: Both datasets achieve their best results when the augmented-positive weight α is around 0.2.For CIFAR-10, ACC drops substantially when α ≥0.35; for CIFAR-100, low values α ≤0.05 perform worse.
  • Default configuration: The default NCL configuration uses |M|=2,000, τ=0.05, k1=|M|/Cu/2, and α=0.2 across datasets.

B.2. Parameter Analysis of HNG

The hard negative generation analysis varies the number of iterations and negative samples against an NCL-only baseline. Every tested setting achieves higher ACC than the baseline without hard negatives.

  • Hard negative generation: All tested values of HNG iterations N and negative samples k2 achieve higher ACC than NCL without hard negative samples.N=0 or k2=0 reduces the model to the NCL-only baseline.

C.1. Different Impact on CIFAR-10 and CIFAR-100

Removing pseudo-positives and adding hard negative generation affect CIFAR-10 and CIFAR-100 differently. The paper attributes this inconsistency to differences in labeled and unlabeled class counts.

  • Class-count effects: The differing pseudo-positive effects are attributed to CIFAR-10 having Cu=5 versus CIFAR-100 having Cu=20.With fewer novel classes, NCL without pseudo-positives treats more positives as negatives in memory for CIFAR-10.
  • Results: NCL reaches 93.4±0.2% on CIFAR-10 and 82.3±2.6% on CIFAR-100, improving over baseline by 5.5% and 12.9%, respectively.
  • Results: Removing pseudo-positives reduces CIFAR-10 ACC to 61.8±7.6% but changes CIFAR-100 ACC only to 68.5±1.9%.
  • Results: Adding HNG yields 93.4±0.1% on CIFAR-10 and 86.6±0.4% on CIFAR-100, corresponding to gains of 5.5% and 17.2% over baseline.
  • Class-count effects: HNG is less diverse on CIFAR-10 because its labeled set has Cl=5, limiting the hard negatives formed by labeled-unlabeled mixing.CIFAR-100 has Cl=80 in the comparison.
Loading 2106.10731v1…