TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

Yanan Wu; Yuhan Yan; Tailai Chen; Zhixiang Chi; ZiZhang Wu; Yi Jin; Yang Wang; Zhenbo Li

Source-linked AI summary

TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

Yanan Wu, Yuhan Yan, Tailai Chen, Zhixiang Chi, ZiZhang Wu, Yi Jin, Yang Wang, Zhenbo Li

arXiv:2603.08075v1Mar 9, 2026cs.CV

TL;DR

OCD must recognize known and discover novel categories in an unlabeled stream, but hash-based methods freeze knowledge and lose feature information. TALON adapts prototypes and the encoder at test time, with offline margin-aware calibration, and consistently surpasses state-of-the-art methods while improving novel-class accuracy and mitigating category explosion.

Problem
Existing OCD methods freeze the feature extractor and use quantized hash prototypes, neglecting incoming-data learning while reducing expressiveness and amplifying intra-class variance.
Method
TALON combines semantic-aware prototype refinement, stable test-time encoder adaptation, continuous feature modeling, and offline margin-aware logit calibration.
Results
TALON consistently surpasses state-of-the-art approaches on standard OCD benchmarks, significantly improving novel-class accuracy and mitigating category explosion.
Takeaways & Limitations
The framework enables knowledge accumulation from streaming samples while maintaining recognition stability and reserving embedding space for future category discovery.
Takeaways & Limitations
The evaluation assumes only the labeled support set is available for training, while query samples arrive sequentially during testing and their labels remain inaccessible during inference.

Abstract

from arXiv · show

On-the-fly category discovery (OCD) aims to recognize known categories while simultaneously discovering novel ones from an unlabeled online stream, using a model trained only on labeled data. Existing approaches freeze the feature extractor trained offline and employ a hash-based framework that quantizes features into binary codes as class prototypes. However, discovering novel categories with a fixed knowledge base is counterintuitive, as the learning potential of incoming data is entirely neglected. In addition, feature quantization introduces information loss, diminishes representational expressiveness, and amplifies intra-class variance. It often results in category explosion, where a single class is fragmented into multiple pseudo-classes. To overcome these limitations, we propose a test-time adaptation framework that enables learning through discovery. It incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update. The former dynamically refines class prototypes to enhance classification, whereas the latter integrates new information directly into the parameter space. Together, these components allow the model to continuously expand its knowledge base with newly encountered samples. Furthermore, we introduce a margin-aware logit calibration in the offline stage to enlarge inter-class margins and improve intra-class compactness, thereby reserving embedding space for future class discovery. Experiments on standard OCD benchmarks demonstrate that our method substantially outperforms existing hash-based state-of-the-art approaches, yielding notable improvements in novel-class accuracy and effectively mitigating category explosion. The code is publicly available at \textcolor{blue}{https://github.com/ynanwu/TALON}.

1. Introduction

On-the-fly category discovery addresses online recognition of known and novel categories, but existing methods neglect learning from incoming data and lose information through hash-based feature quantization. TALON adapts prototypes and the encoder at test time while calibrating the offline embedding space for future discovery.

OCD recognizes known categories and discovers novel ones from an unlabeled online stream using a model trained only on labeled data.
Existing OCD methods freeze the offline feature extractor and quantize image features into binary hash codes for class prototypes.
Hash-based quantization reduces feature expressiveness, limits prototype information, and increases intra-class variation, causing category explosion.Category explosion fragments one actual class into multiple pseudo-classes.
TALON updates both model parameters and class prototypes during test time to absorb information from incoming samples under semantic shift.Its prototype update uses confidence-controlled exponential moving averages, while encoder adaptation uses entropy and prototype-level regularization.
Margin-aware logit calibration improves intra-class compactness and inter-class separability, reserving embedding space for emerging novel categories.
Experiments on standard benchmarks show that TALON surpasses existing state-of-the-art methods and mitigates category explosion through gains on novel classes.

2. Related Work

Prior category-discovery settings progressively relax assumptions about unlabeled data, but they remain tied to offline training or inference. OCD instead targets streaming, instance-level feedback without a predefined query set, while TTA adapts models to test-time shifts.

Category Discovery: NCD discovers novel classes in unlabeled data using knowledge transferred from labeled examples, while assuming all unlabeled samples are novel.
Category Discovery: GCD allows unlabeled data to mix known and novel categories, improving flexibility over NCD while still relying on offline training.
Category Discovery: Continual GCD processes multiple sequential unlabeled stages, but its offline inference limits applicability when data streams continuously and immediate feedback is required.
On-the-Fly Category Discovery: OCD removes the predefined query-set assumption and supports instance-level feedback in streaming data through a hash-code-based framework.
Test-Time Adaptation: TTA adapts trained models toward test data to mitigate degradation caused by semantic shifts, including approaches that optimize unsupervised objectives or prediction entropy.

3. The Proposed Method

TALON formulates OCD as online test-time adaptation: it learns from sequential unlabeled queries while recognizing known classes and expanding prototypes for novel categories. Its hash-free representation learning combines margin-aware offline calibration with semantic-aware prototype refinement and stable entropy-based encoder adaptation.

Problem definition: OCD identifies known and novel categories from sequential unlabeled queries using only labeled known-class data for offline training.Query labels are inaccessible during inference and used only for evaluation.
Methods overview: TALON introduces a hash-free TTA framework that continually adapts model parameters and class prototypes to semantic shifts during inference.The framework is designed for novel-category emergence rather than only conventional domain shift.
Representation learning on labeled data: Margin-aware logit calibration enlarges inter-class separation and tightens intra-class compactness by applying an angular margin to normalized features and class weights.The class-weight columns of the projector represent known-category prototype directions in the embedding space.
Online recognition and novelty handling: During online inference, samples are compared with cosine-similarity prototypes, and low-similarity samples create new prototypes for novel categories.The prototype memory therefore expands sequentially while recognizing samples assigned to existing categories.
Online adaptation: Semantic-aware prototype updates use confidence and support size to control exponential-moving-average refinement, while entropy minimization and prototype regularization stabilize encoder adaptation.Updates are conservative when data are scarce and preserve alignment between features and their corresponding prototypes.

4. Experiment

Experiments across seven benchmark datasets and multiple evaluation protocols show that TALON consistently outperforms prior methods while improving novel-category discovery and mitigating category explosion. Ablations and analyses attribute these gains to margin calibration, prototype refinement, encoder adaptation, and hash-free continuous features.

Dataset and evaluation: Experiments cover seven coarse- and fine-grained benchmark datasets and use Greedy-Hungarian and Strict-Hungarian evaluation protocols.Greedy-Hungarian separately evaluates Old and New classes, whereas Strict-Hungarian evaluates the entire query set.
Ablation study: MLC, prototype refinement, and encoder adaptation each improve performance, while their combination produces the highest overall results on CUB and SCars.MLC strengthens the offline feature space; TTA-P updates prototypes; TTA-M adapts encoder parameters during testing.
Category discovery: Hash-free continuous feature modeling mitigates category explosion and supports accurate, stable category discovery, unlike hash-based methods whose category estimates depend strongly on code length.Larger hash lengths can severely overestimate category numbers, while shorter codes reduce discriminability, especially for novel classes.
Comparison with test-time adaptation methods: Conventional TTA methods provide only marginal gains or can degrade performance because they target domain shift with fixed label spaces rather than semantic shift requiring novel-category discovery.TALON instead adapts both prototype memory and encoder parameters.
Ablation study: MLC reduces the mean sample–prototype angle from 64.55° to 35.83° and increases the mean inter-prototype angle from 27.98° to 74.15° on Pets.The analysis links these changes to stronger intra-class compactness and inter-class separability.
Hyperparameter analysis: Larger adaptation batches improve performance before saturation, moderate angular margins provide the best trade-off, and higher similarity thresholds yield more newly discovered categories with minor accuracy impact.Figure 4 reports accuracy for All, Old, and New classes alongside the number of newly discovered categories.

5. Conclusion

TALON addresses information loss and category explosion in OCD by combining test-time knowledge accumulation with margin-aware offline calibration. Experiments show consistent gains in novel-class accuracy while mitigating category explosion.

TALON combines semantic-aware prototype refinement with stable encoder adaptation to absorb information from streaming data while maintaining recognition stability.
Margin-aware Logit Calibration enlarges inter-class margins and reserves embedding space for future category discovery.
TALON consistently surpasses state-of-the-art OCD approaches, improving novel-class accuracy and mitigating category explosion.

A.1. Datasets Details and Evaluation Metric Details

The evaluation uses diverse coarse- and fine-grained datasets with seen and unseen category splits. Performance is assessed with Greedy-Hungarian and Strict-Hungarian protocols based on cluster assignments and Hungarian matching.

Datasets Details: Experiments cover coarse-grained and fine-grained recognition scenarios using the datasets listed in the benchmark statistics tables.
Evaluation Metric Details: Greedy-Hungarian and Strict-Hungarian provide complementary evaluation protocols for OCD cluster assignments.
Evaluation Metric Details: Greedy-Hungarian reports Old and New accuracy separately, whereas Strict-Hungarian evaluates the entire query set without repeated assignments between subsets.
Evaluation Metric Details: The overall accuracy is obtained through Hungarian matching between predicted cluster labels and ground-truth label permutations.

A.2. Training Details

Training and evaluation use multiple visual backbones, optimization settings, and comparison methods. Additional analyses examine hashing baselines, hyperparameters, backbone robustness, and performance across benchmarks.

Training Details: TALON is evaluated with CLIP ViT-B/16 and DINOv2-ViT-Base pretrained backbones, producing 512- and 768-dimensional visual representations, respectively.
Training Details: Optimization uses AdamW with weight decay 0.05, cosine scheduling to 1e−5, batch size 128, and parameter-specific learning rates.
Baselines: The study reimplements Ranking Statistics and compares WTA, SLC, MLDG, SMILE, PHE, and DiffGRE under OCD evaluation settings.
Hyperparameter Analysis: The logit scale s controls MLC logit magnitude, while smoothing constant κ controls how quickly prototypes adapt to incoming test samples.
Backbone Comparison: TALON consistently outperforms SMILE and PHE across both CLIP and DINO backbones and all benchmarks, especially on New categories.
Backbone Comparison: The backbone experiments validate TALON’s stability across diverse representation settings.

B.3. Computational Complexity Analysis.

The computational analysis separates OCD into training, prototype computation, and testing stages. Testing consumes the largest time proportion because of test-time adaptation.

Computational Complexity Analysis: The OCD pipeline comprises training, prototype computation, and testing stages, measured across five runs using CLIP on one NVIDIA RTX 3090 GPU.
Computational Complexity Analysis: Testing has the largest time proportion among the three stages because TALON introduces a test-time adaptation algorithm.

B.4. Training Efficiency Analysis

TALON is evaluated for training efficiency under a controlled comparison and achieves lower training times than SMILE and PHE across four datasets. The efficiency advantage is attributed to its feature-learning strategy and reduced augmentation overhead.

Experimental Setup: Training times are compared under identical GPU, epoch, mixed-precision, dataloader, and batch-size settings.All methods are trained for 100 epochs on an NVIDIA RTX 3090 with batch size 128.
Results: Table 9 reports consistently lower training time for TALON than both SMILE and PHE across all four datasets.The table compares training times in seconds.
Results: 2752.6 s, 4023.5 s, 14843.2 s, and 1572.0 s are TALON’s reductions relative to SMILE on CUB, SCars, Food, and Pets, respectively.Compared with PHE, reductions are 1847.7 s, 3490.4 s, 10608.2 s, and 1213.4 s on the same datasets.
Interpretation: The lower training time is attributed to more efficient feature learning, while SMILE and PHE add contrastive-learning, estimation, or augmentation overhead.SMILE processes two augmented views per sample for supervised contrastive learning.

C.2. Limitations.

The framework depends on strong pretrained backbones and GPU resources, and its evaluation scope is limited mainly to standard OCD benchmarks with moderate numbers of novel classes. Future deployment therefore motivates lighter backbones and stronger handling of long-term distribution shift.

Limitations: Performance may degrade in low-resource scenarios or when CLIP and DINOv2 pretraining is unavailable.The framework relies on strong pretrained vision backbones and GPU resources.
Limitations: Evaluation primarily covers standard OCD benchmarks with moderate numbers of novel classes, leaving more extreme streams less established.Very large numbers of emerging categories may increase prototype memory and adaptation instability.
Future Work: Future work proposes lightweight or distilled backbones, stronger long-term-shift handling, and multimodal or human feedback for category management.Suggested mechanisms include memory-based replay and more robust prototype regularization.