Source-linked AI summary
CloSeR: Unified Relational Distillation from Closed-Set Teachers for Category Discovery
Yuanpei Liu, Zhenqi He, Jialu Tang, Kai Han
TL;DR
GCD must recognize known classes and discover novel categories from partially labelled data, but one-stage mixed-objective training can conflict and distort pretrained representations. CloSeR builds an adapter-based closed-set teacher and transfers global and local relational structure through URD, yielding consistent gains across six benchmarks with DINO and DINOv2.
Problem
GCD seeks to recognize known classes and discover novel categories from partially labelled data, while existing mixed-objective training can conflict and distort pretrained representations.
Method
CloSeR tunes lightweight block-wise adapters on labelled data to build a closed-set teacher, then uses URD to distill global prototype and local neighborhood relations through decoupled pathways.
Results
CloSeR yields consistent improvements across parametric and non-parametric GCD baselines on six benchmarks using DINO and DINOv2 backbones.
Takeaways & Limitations
CloSeR provides a head-agnostic relational regularizer with a favorable accuracy–efficiency trade-off and improved use of larger backbones.
Takeaways & Limitations
The GCD setup assumes the total number of categories K is given, although existing techniques can estimate K when it is unknown.
Abstract
from arXiv · showhide
Generalized Category Discovery (GCD) is an intriguing open-world problem that has garnered increasing attention: given partially labelled data, the goal is to correctly recognize known classes while discovering coherent novel categories from unlabelled samples. Recent GCD methods typically adapt foundation models by jointly optimizing supervised classification and unsupervised discovery objectives on mixed labelled and unlabelled data. While effective, this coupled training can entangle closed-set recognition and open-set discovery, leading to objective conflict and biased predictions, and may disturb the semantic geometry of pretrained representations under limited labels and noisy pseudo-labels. We propose CloSeR, a simple plug-and-play framework that injects Closed-Set Relational knowledge into GCD training. CloSeR first builds a domain-adapted closed-set teacher by tuning lightweight block-wise adapters on labelled known-class data while keeping the foundation model backbone frozen, thereby preserving pretrained priors at low training cost. It then transfers the teacher's knowledge to downstream GCD via Unified Relational Distillation (URD), which distills complementary global sample-to-prototype relations to anchor known-class semantics and local sample-to-sample relations to preserve neighborhood structure, using separate feature pathways to reduce optimization interference. CloSeR is head-agnostic and readily integrates with both parametric and non-parametric GCD methods. Extensive experiments with DINO and DINOv2 backbones on six benchmarks (CIFAR-10/100, ImageNet-100, CUB, Stanford-Cars, and FGVC-Aircraft) show consistent gains over GCD baselines, achieving state-of-the-art performance. Project page: https://visual-ai.github.io/closer/
1 Introduction
Generalized Category Discovery combines known-class recognition with novel-category discovery from partially labelled data. CloSeR addresses conflicts in one-stage training by adapting a closed-set teacher first and transferring its relational knowledge during GCD.
- GCD recognizes known classes and groups unlabelled samples from unknown categories into coherent novel classes.
- Recent methods jointly optimize supervised and unsupervised objectives while adapting foundation-model backbones on mixed labelled–unlabelled data.
- This one-stage coupling may create optimization conflict, prediction bias, and distortion of pretrained feature geometry under limited labels and noisy pseudo-labels.
- A pilot study finds that adapting the backbone on labelled known classes before k-means or parametric GCD improves both non-parametric and parametric alternatives over standard GCD baselines.
- CloSeR builds a domain-adapted closed-set teacher with lightweight block-wise adapters, then distills global prototype relations and local neighborhood relations through separate feature pathways.
- CloSeR integrates with both parametric and non-parametric GCD methods and shows consistent gains across six benchmarks with DINO and DINOv2 backbones.
2 Related Work
Category discovery research spans transfer-based novel discovery and generalized discovery, while self-supervised learning provides a common foundation. CloSeR contributes relational distillation that transfers closed-set structure to heterogeneous GCD students.
- Generalized Category Discovery extends Novel Category Discovery by allowing unlabelled data to contain both known and novel categories.
- Self-supervised learning and self-distillation are common foundations for modern category-discovery methods.
- CloSeR distills both global and local relations in decoupled feature spaces to inject closed-set categorical priors into GCD models.
- The relational objective transfers teacher-induced structure to both parametric and non-parametric GCD methods.
3 Preliminaries
GCD jointly classifies unlabelled known samples and clusters unlabelled unknown samples, using supervised and unsupervised objectives in common baselines. These baselines can entangle objectives and distort pretrained geometry.
- Problem Setup and Notation: GCD jointly classifies unlabelled samples from known classes and partitions the remaining unlabelled samples into unknown-category clusters.
- Problem Setup and Notation: The unlabelled pool contains known and novel categories, with the known label space contained in the unlabelled label space.
- Problem Setup and Notation: The setup assumes the total number of categories K is given, although existing techniques can estimate K when it is unknown.
- Baselines: SimGCD learns a parametric classifier through self-distillation with normalized prototypes and temperature-scaled feature–prototype similarities.
- Limitation of Baselines: Baseline methods jointly optimize supervised and unsupervised objectives while fine-tuning the backbone on mixed labelled–unlabelled data.
- Limitation of Baselines: This coupling can conflict with closed-set discrimination, bias representations toward known categories, and distort foundation-model geometry for transfer clustering.
4 Method
CloSeR uses staged closed-set transfer and unified relational distillation to inject adapted semantic relations into GCD training. Its teacher anchors samples to known-class prototypes and preserves local neighborhood structure while the student retains the standard GCD objective.
- Closed-Set Transfer Learning: CloSeR first adapts a frozen foundation model with lightweight block-wise adapters and learns closed-set prototypes from labelled data.Adapters update features across transformer blocks while preserving the original foundation weights.
- Closed-Set Transfer Learning: The closed-set transfer model produces ℓ2-normalized features and uses M normalized prototypes as a cosine-classifier head.Normalization matches cosine-similarity classification and stabilizes transfer to the target domain.
- Unified Relational Distillation: During GCD training, URD uses a frozen closed-set teacher and a student initialized from the same foundation model, with only selected student blocks and GCD heads fine-tuned.A lightweight mapping network aligns student features with the teacher’s closed-set space without directly forcing feature imitation.
- Unified Relational Distillation: Global URD distills each sample’s similarity distribution over fixed closed-set prototypes through KL divergence.This preserves the teacher’s relative similarity profile to known-class anchors while allowing capacity for novel categories.
- Unified Relational Distillation: Local URD converts temperature-scaled sample-to-sample similarities within mixed batches into row-stochastic distributions and matches teacher and student neighborhoods with KL divergence.The local term targets fine-grained relational geometry, including intra-class compactness and inter-class separation.
- Unified Relational Distillation: CloSeR optimizes the standard GCD loss together with weighted global and local URD losses and is evaluated with both parametric and non-parametric baselines.The framework is illustrated with SimGCD and its results are reported for All, Old, and New categories on generic datasets.
5 Experiments
Experiments evaluate CloSeR with DINO and DINOv2 across generic and fine-grained benchmarks, parametric and non-parametric baselines, transfer strategies, distillation components, backbone sizes, and attention maps. CloSeR delivers consistent gains, with especially large improvements on fine-grained datasets and evidence that stronger transfer and teachers benefit discovery.
- Quantitative Comparison: CloSeR is evaluated with DINO and DINOv2 on six generic and fine-grained benchmarks using SelEx and SimGCD baselines.Performance is measured with clustering accuracy across All, Old, and New categories.
- Quantitative Comparison: Under DINO, CloSeR raises SimGCD average All accuracy from 56.1 to 62.6 on fine-grained datasets, while Cars All increases from 53.8 to 63.0.The Cars improvement is +9.2 points, and the average improvement is reported as +11.6% relative.
- Quantitative Comparison: Across all six datasets and both backbones, CloSeR consistently improves Old and New category performance across heterogeneous GCD pipelines.The discussion attributes this pattern to mitigating common training bias and preserving transferable foundation-model structure.
- Diagnostic Study: Block-wise adapter adaptation supports shallow-to-deep feature transfer, whereas tuning only the last transformer block is insufficient under weaker DINO pretraining.The comparison includes full fine-tuning, last-block tuning, and block-wise adapters.
- Diagnostic Study: A ViT-L teacher improves CUB All accuracy from 78.1 to 81.3 with a fixed ViT-B student, indicating benefits from stronger teachers.The ablation also examines global relations, local relations, and feature decoupling in URD.
- Diagnostic Study: Attention maps compare SimGCD and SimGCD+CloSeR across Stanford-Cars, CUB, and FGVC-Aircraft for both Old and New classes.The maps are derived from the last transformer block, averaged over heads, upsampled, and normalized to [0, 1].
6 Conclusion
CloSeR improves generalized category discovery through staged closed-set transfer and unified relational distillation. The framework preserves pretrained knowledge while adapting shallow-to-deep features and transfers both global prototype relations and local neighborhood structure.
- Conclusion: CloSeR combines closed-set transfer learning with unified relational distillation to improve generalized category discovery.The framework is presented as a staged approach leveraging foundation models.
- Conclusion: Block-wise adapter tuning builds a domain-adapted closed-set teacher while preserving pretrained knowledge and efficiently adapting shallow-to-deep features.The foundation-model backbone remains frozen during this transfer stage.
- Conclusion: URD transfers prototype-anchored global relations and batch-level local neighborhood structure with feature decoupling to reduce optimization interference.These relations are distilled from the closed-set space into the GCD student.
- Conclusion: Experiments and ablations show consistent gains over strong baselines, a favorable accuracy–efficiency trade-off, and improved exploitation of model scaling.The conclusion characterizes this as unlocking the potential of larger backbones for category discovery.