PaperlayerView exact revision on arXiv

Source-linked AI summary

Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation

Bingchen Zhao, Kai Han

arXiv:2107.03358v2cs.CV

TL;DR

Novel category discovery must partition unlabelled images from new classes using knowledge from labelled but different relevant classes, a setting that is harder than conventional semi-supervised learning. The paper combines global and local branches with dual ranking statistics and mutual knowledge distillation, and reports state-of-the-art performance across generic and fine-grained benchmarks. Its scope includes the assumption that labelled and unlabelled classes are relevant though different, with the number of unlabelled classes known or estimated separately.

  • Problem

    Novel category discovery asks how to partition unlabelled images from new classes by leveraging labelled images from relevant but different categories, without assuming labelled examples for every unlabelled class.

  • Method

    The method uses global and local branches, dual ranking statistics for pseudo-label transfer, and mutual knowledge distillation to exchange information and encourage agreement.

  • Results

    The method achieves state-of-the-art performance on generic object-classification benchmarks and challenging fine-grained recognition datasets.

  • Takeaways & Limitations

    Combining global characteristics with local part-level information supports more reliable novel category discovery across generic and fine-grained datasets.

  • Takeaways & Limitations

    The approach assumes that labelled and unlabelled classes are relevant though different, and assumes the number of unlabelled classes is known or estimated separately.

Abstract

from arXiv · show

In this paper, we tackle the problem of novel visual category discovery, i.e., grouping unlabelled images from new classes into different semantic partitions by leveraging a labelled dataset that contains images from other different but relevant categories. This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from the labelled data to the unlabelled, we propose using dual ranking statistics on both branches to generate pseudo labels for training on the unlabelled data. We further introduce a mutual knowledge distillation method to allow information exchange and encourage agreement between the two branches for discovering new categories, allowing our model to enjoy the benefits of global and local features. We comprehensively evaluate our method on public benchmarks for generic object classification, as well as the more challenging datasets for fine-grained visual recognition, achieving state-of-the-art performance.

1 Introduction

Novel category discovery transfers class knowledge from labelled categories to partition unlabelled images from new categories, addressing annotation limits and the gap between standard semi-supervised learning and truly novel classes. The paper uses complementary local and global branches, dual ranking statistics, and mutual knowledge distillation, achieving state-of-the-art results on generic and fine-grained benchmarks.

  • Motivation: Large-scale annotation is difficult because real-world classes continually expand and some domains require specialized expertise.The paper cites ImageNet-scale annotation and medical data as examples of this challenge.
  • Motivation: Novel category discovery transfers knowledge from labelled data containing relevant but different classes to partition unlabelled data into new visual categories.Unlike semi-supervised learning, it does not assume that every unlabelled class has labelled instances.
  • Approach: The framework combines a local part-level branch with a global branch focused on overall characteristics.This design targets complementary information for discovering new categories.
  • Approach: Dual ranking statistics transfer knowledge from known classes by generating pseudo labels from both part-level and global similarity information.The method maintains a dynamic object-part dictionary for local ranking statistics while also using global descriptors.
  • Approach: Mutual knowledge distillation exchanges information between the local and global branches and encourages their agreement without requiring manual annotations.The two branches therefore benefit from one another during novel-category discovery.
  • Evaluation: State-of-the-art results were obtained on generic datasets including CIFAR10, CIFAR100, and ImageNet, as well as fine-grained CUB-200, Stanford-Cars, and FGVC-Aircraft.The authors attribute the fine-grained gains to exploiting both local and global information.

2 Related work

Novel category discovery uses relevant labelled categories to impose a class prior when partitioning unlabelled data, distinguishing it from both unsupervised clustering and semi-supervised learning. The paper relates this setting to knowledge distillation and part-level feature learning while addressing the omission of local object parts in prior methods.

  • Novel category discovery: Novel category discovery learns a class prior from labelled but different relevant categories to define partitions among unlabelled data.This distinguishes it from unsupervised clustering, which can produce multiple valid partitions under different criteria.
  • Novel category discovery: Unlike semi-supervised learning, novel category discovery does not assume that every unlabelled class has labelled instances.The distinction makes it a separate setting despite their shared use of labelled and unlabelled data.
  • Novel category discovery: Prior novel-category-discovery methods considered global descriptors while ignoring local object parts that help distinguish visually similar classes.This paper jointly models both information sources and permits information exchange between them.
  • Knowledge distillation: Knowledge distillation commonly enforces agreement between teacher and student outputs or representations, while related self-distillation and mutual-learning methods exchange knowledge between models.The reviewed methods generally require labels, limiting their direct use for novel category discovery.
  • Part-level feature learning: Part-level features have supported image verification tasks such as few-shot learning and image retrieval by providing local visual matching primitives.Prior work generated part-level concepts by clustering feature-map vectors for matching at test time.

3 Method

The method discovers novel categories with global and local branches that transfer labelled-data knowledge through dual ranking statistics and mutual distillation. Global descriptors and local parts generate complementary pseudo-labels, while branch agreement is encouraged through similarity-distribution matching.

  • 3 Method: The framework jointly uses global image-level and local part-level branches with shared feature extraction, separate projections and heads, and pseudo-label-based learning for labelled and unlabelled categories.The global branch captures overall features, while the local branch focuses on individual spatial parts.
  • 3.3 Overall training loss: The method combines binary cross-entropy, mutual-distillation, supervised cross-entropy, and consistency losses to train both branches.Consistency regularization prevents augmented versions of the same image from receiving different ranking-statistics pseudo-labels.
  • 3.1 Dual ranking statistics for knowledge transfer: Global ranking statistics compare top-k elements of unlabelled image embeddings to produce pair-wise pseudo-labels trained with binary cross-entropy.The pseudo-label measures overlap between the top-k ranks of two global feature vectors.
  • 3.1 Dual ranking statistics for knowledge transfer: Local ranking statistics compare each object part with a FIFO memory bank of part descriptors, average-pool the resulting similarity vectors, and generate pair-wise pseudo-labels.The memory bank acts as a dynamic object-part dictionary containing sampled spatial features.
  • 3.2 Mutual knowledge distillation for novel category discovery: Mutual knowledge distillation exchanges information between the global and local branches by matching their similarity-score distributions over separate feature banks with symmetric KL divergence.Unlike teacher–student distillation, both branches of the same model are jointly trained while retaining distinct global and local focuses.

4 Experimental results

Experiments evaluate the method across generic and fine-grained novel category discovery benchmarks, then analyze its components, branch configurations, memory banks, sampling strategy, and unknown-class settings.

  • Comparison to the state-of-the-art: 6.4% improvement over RankStat on ImageNet-1K establishes state-of-the-art performance on generic classification datasets.The method also outperforms k-means, KCL, MCL, and DTC; gains on CIFAR-10 and CIFAR-100 are smaller.
  • Comparison to the state-of-the-art: 8.3%, 8.1% and 4.1% improvements over RankStat on CUB-200, Stanford-Cars and FGVC-Aircraft establish a new state of the art.The full method outperforms global-feature methods, while the local branch alone already improves over RankStat.
  • Ablation study: Removing sKLD drops ACC by 8.0–11.3% and removing consistency drops it by 9.9–12.2%, while removing BCE makes the local-branch parameters effectively random.The full method performs best when all components are retained.
  • Ablation study: Global-local mutual distillation performs best among the tested two-branch configurations, outperforming single-branch and same-type branch alternatives.Local single-branch configurations outperform global ones, and adding a second branch consistently improves performance.
  • Ablation study: Increasing memory-bank sizes generally improves performance, although Bp and Bg gains plateau quickly on CUB-200 and Stanford-Cars.The V bank improves consistently across datasets; CAM-selected parts outperform random selection, while using all parts introduces redundancy.
  • Ablation study: Using estimated novel-class counts with DTC still yields significantly better performance than RankStat on CUB-200 and ImageNet-100.This evaluates the method when the number of unlabelled classes is unknown.

5 Conclusion

The paper presents a two-branch framework for novel category discovery that combines local part-level and global information. It transfers knowledge through ranking statistics and mutual distillation, achieving state-of-the-art performance across generic and fine-grained recognition datasets.

  • The framework combines local part-level and global branches for novel category discovery.
  • Local ranking statistics use an on-the-fly part dictionary alongside global ranking statistics to transfer knowledge from labelled to unlabelled data.
  • Mutual knowledge distillation exchanges information between branches and encourages their agreement.
  • The method achieves state-of-the-art performance on generic image classification and fine-grained recognition datasets.

–Supplementary Material–

The supplementary material accompanies the NeurIPS 2021 paper and identifies its authors and arXiv version.

  • The listed authors are Bingchen Zhao and Kai Han.
  • The arXiv version is 2107.03358v2, dated 3 January 2022.
  • The paper was published at NeurIPS 2021.

A Implementation details

The experiments initialize models with pretrained self-supervised representations and use dataset-dependent training schedules.

  • MoCo v2 pretrained for 800 epochs on ImageNet-1K initializes experiments except CIFAR10/100 and ImageNet-1K.
  • CIFAR10/100, CUB-200, Stanford-Cars, FGVC-Aircraft, and ImageNet-100 use 200 training epochs, while ImageNet-1K uses 90.

B Unlabelled data containing both seen and unseen classes

The supplementary experiments address open-world data containing both seen and novel classes and examine ranking-statistics robustness to the choice of k. Results are generally stable except at very small k, with defaults of 5 globally and 30 locally.

  • B Unlabelled data containing both seen and unseen classes: The open-world setting requires recognizing seen classes while discovering novel classes in unlabelled data.
  • B Unlabelled data containing both seen and unseen classes: The model is extended for open-world data by enlarging the unlabelled classification head to include seen and novel classes.
  • B Unlabelled data containing both seen and unseen classes: Performance is generally stable across k values for global and local ranking statistics, except when k is extremely small, such as k = 1.
  • B Unlabelled data containing both seen and unseen classes: The default k values are 5 for global ranking statistics and 30 for local ranking statistics.

D A single-branch variant of our method with dual ranking statistics

The single-branch variant uses local ranking statistics for positive pairs and global ranking statistics for negative pairs, based on their differing reliability in pairwise verification.

  • D A single-branch variant of our method with dual ranking statistics: Local ranking statistics provide more reliable positives, whereas global ranking statistics provide more reliable negatives in pairwise verification.This pairing motivates the single-branch variant, which does not use mutual learning.

E Qualitative results

t-SNE visualizations show that unlabelled-data features become increasingly discriminative during training on ImageNet-100 and Stanford Cars.

  • E Qualitative results: Features on unlabelled data become more discriminative throughout training for both ImageNet-100 and Stanford Cars.MoCo v2 initialization appears better for ImageNet-100, consistent with ImageNet-100 being a subset of its ImageNet-1K pretraining data.

F Comparing ranking statistics with cosine similarity

The supplementary comparisons examine ranking statistics against cosine similarity, alternative training stages, and mutual-learning losses across the model’s design choices.

  • F Comparing ranking statistics with cosine similarity: The t-SNE visualization tracks feature evolution for ImageNet-100 and Stanford Cars during training.Rows correspond to the two datasets, with data-point colors denoting ground-truth labels.
  • F Comparing ranking statistics with cosine similarity: Hard and soft modes perform comparably for both ranking statistics and cosine similarity in pseudo-label generation.The comparison includes hard ranking statistics, soft ranking statistics, hard cosine similarity with a 0.9 threshold, and soft cosine similarity.
  • F Comparing ranking statistics with cosine similarity: Additional supervised finetuning after self-supervised pretraining does not bring obvious gains for the model.The added stage freezes the first three ResNet macro blocks and finetunes the last macro block on labelled categories before joint training.
  • F Comparing ranking statistics with cosine similarity: Symmetric Kullback-Leibler divergence and Jensen-Shannon divergence are both valid losses for mutual learning in this model.The paper reports that both losses perform equally validly in the comparison.

I Limitations and potential negative societal impacts

The method’s public-dataset performance remains below fully supervised models, and its reliability is uncertain for complex, safety-critical real-world settings.

  • I Limitations and potential negative societal impacts: Performance still notably lags behind fully supervised models despite achieving state-of-the-art results on public datasets.The limitation is stated alongside the need to validate the method for specific deployment scenarios.
  • I Limitations and potential negative societal impacts: Reliable inference is not expected in safety-critical applications when unlabelled data differs substantially from labelled data or contains unpredictable noise.The paper specifically cautions against relying on the method in autonomous driving and medical image analysis without careful validation.
  • I Limitations and potential negative societal impacts: The experiments use datasets permitted for research or non-commercial research and educational purposes.The stated dataset access terms include MIT licensing for CIFAR-10 and CIFAR-100 and research-use restrictions for several other datasets.
Loading 2107.03358v2…