Source-linked AI summary
Learning Placeholders for Open-Set Recognition
Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan
TL;DR
Open-set recognition must preserve known-class classification while detecting unseen categories that closed-set models may classify as known with overconfident scores. PROSER addresses this by learning data and classifier placeholders; experiments report improved unknown detection and open-set recognition across datasets, including a 6% gain over GFROSR on CIFAR10.
Problem
Closed-set classifiers mishandle unseen categories because they lack novel-class data and produce overconfident, composition-sensitive confidence scores, making calibration difficult.
Method
PROSER anticipates novel classes with data placeholders and adds classifier placeholders that adaptively separate known from unknown classes.
Results
PROSER improves unknown detection on natural-image datasets, including 6% over GFROSR on CIFAR10, while preserving closed-set discriminative ability.
Takeaways & Limitations
Placeholder-based calibration supports open-set recognition by preparing both the training data and classifier for unknown categories.
Abstract
from arXiv · showhide
Traditional classifiers are deployed under closed-set setting, with both training and test classes belong to the same set. However, real-world applications probably face the input of unknown categories, and the model will recognize them as known ones. Under such circumstances, open-set recognition is proposed to maintain classification performance on known classes and reject unknowns. The closed-set models make overconfident predictions over familiar known class instances, so that calibration and thresholding across categories become essential issues when extending to an open-set environment. To this end, we proposed to learn PlaceholdeRs for Open-SEt Recognition (Proser), which prepares for the unknown classes by allocating placeholders for both data and classifier. In detail, learning data placeholders tries to anticipate open-set class data, thus transforms closed-set training into open-set training. Besides, to learn the invariant information between target and non-target classes, we reserve classifier placeholders as the class-specific boundary between known and unknown. The proposed Proser efficiently generates novel class by manifold mixup, and adaptively sets the value of reserved open-set classifier during training. Experiments on various datasets validate the effectiveness of our proposed method.
1. Introduction
Open-set recognition addresses the failure of closed-world classifiers on unseen categories. PROSER prepares models for this setting with data and classifier placeholders to improve calibration.
- Closed-set models can classify unseen-category instances as known classes because training cannot cover all categories.
- Open-set recognition aims to classify known instances while detecting those from unknown classes.
- Thresholding unknowns is difficult because deep models produce overconfident probabilities and unknown categories have diverse distributions.
- PROSER learns data placeholders that anticipate novel classes and classifier placeholders that provide class-specific boundaries between known and unknown.
2. Related Work
Related work approaches open-set recognition through discriminative or generative models, while out-of-distribution and novelty detection address related but narrower detection problems.
- Open-Set Recognition: Discriminative open-set methods include traditional SVM- and distance-based approaches as well as deep learning methods.
- Open-Set Recognition: Generative approaches synthesize or construct novel instances, including unknown samples, boundary examples, and class-conditioned reconstructions.
- Out-Of-Distribution Detection: Out-of-distribution, anomaly, and novelty detection are related to open-set recognition but generally formulate unseen-class detection as binary classification.
3. From Closed-set to Open-set Recognition
Closed-set classifiers assume training and test data share a distribution and classify among known classes. Extending them to open-set recognition requires detecting novel categories despite overconfident and composition-sensitive predictions.
- 3.1. Closed-Set Classification: Closed-set classification trains on labeled instances from K known classes and assumes training and test data are drawn from the same distribution.
- 3.1. Closed-Set Classification: The classifier represents inputs with an embedding and linear class weights, typically optimized with cross-entropy for discrimination among known classes.
- 3.2. Open-Set Recognition: Open-set test data include known classes and a novel-class group labeled K + 1, which may contain multiple unknown categories.
- 3.2. Open-Set Recognition: A closed-set classifier predicts unknown instances as known classes because it has not observed open-set examples during training.
- 3.2. Open-Set Recognition: Thresholding uses maximum known-class probability as confidence, but separating known and unknown instances is hard because both confidences can approach 1.
- 3.2. Open-Set Recognition: Threshold selection also depends on similarity between known and unknown categories, which varies across known-unknown compositions.
4. Learning Placeholders for Open-Set Recognition
PROSER prepares closed-set classifiers for open-set recognition by learning placeholders for both unknown-class data and classifier boundaries. It combines manifold mixup with dummy classifiers and calibration to tighten decision boundaries while preserving known-class classification.
- PROSER uses data placeholders to anticipate novel classes and transform closed-set training into open-set training.
- Classifier placeholders add dummy classifiers that separate known from unknown instances using class-specific, input-dependent thresholds.The dummy classifier shares the embedding with the closed-set classifier and is trained to output the second-largest probability for known instances.
- The classifier-placeholder loss preserves correct known-class predictions while placing the dummy classifier between target and non-target classes.Its second term matches masked probabilities to the extra classifier, making the dummy output the second-largest probability.
- Manifold mixup combines different-class examples at a middle hidden layer to create data placeholders in low-confidence regions near decision boundaries.Mini-batch shuffling and masking same-class pairs keep the computation complexity at the same magnitude as vanilla training.
- Data placeholders tighten known-class embeddings and push the decision boundary toward mixed composition classes, leaving more space for novel classes.Figure 2 illustrates classifier placeholders between target and non-target classes and data placeholders near the decision-boundary manifold.
- PROSER further calibrates dummy logits with validation-set bias tuning, selecting the bias so 95% of validation data is recognized as known.The validation set is drawn from the same distribution as the training set and provides no access to unknown data.
5. Experiment
Experiments evaluate PROSER for unknown detection, open-set recognition, ablations, and decision-boundary behavior across several datasets. PROSER improves unknown detection and open-set recognition while preserving closed-set discrimination, with both data and classifier placeholders contributing to calibration.
- Unknown Detection: Experiments evaluate unknown detection using mean AUC, with results averaged across five randomized trials and benchmark datasets spanning different openness levels.SVHN and CIFAR10 use 22.54% openness, CIFAR+10 and CIFAR+50 use 46.55% and 72.78%, and Tiny-ImageNet uses 68.37%.
- Unknown Detection: 6% improvement over GFROSR on CIFAR10 and 4% improvements on CIFAR+10, CIFAR+50, and Tiny-ImageNet demonstrate stronger unknown detection on natural-image datasets.The results on SVHN are described as nearly saturated, while PROSER does not significantly degrade closed-set accuracy.
- Open-Set Recognition: PROSER achieves the best reported open-set recognition performance on MNIST across outliers from Omniglot, MNIST-noise, and Noise.The evaluation uses macro F1 over ten known classes and one unknown class with a 1:1 known-unknown test ratio.
- Open-Set Recognition: PROSER achieves better macro F1 than state-of-the-art methods on CIFAR10 with diverse outlier inputs resized or cropped to 32*32.The evaluation uses ten known classes and one unknown class, with equal numbers of known and novel test instances.
- Ablation Study: As openness increases, macro F1 declines, but manifold mixup and dummy classifiers each improve plain CNN, while combining both components performs best.Using more than one dummy classifier also improves performance, indicating that classifier-placeholder diversity matters.
- Visualization of Decision Boundaries: Traditional fixed thresholds that separate MNIST known classes from one novel class fail to transfer to another, whereas PROSER learns adaptive instance-specific thresholds.Data and classifier placeholders provide invariant information between target and non-target classes and support changing novel-class distributions.
6. Conclusion
PROSER addresses open-set recognition by preparing both data and classifiers for unseen classes. It uses manifold mixup and adaptive reserved classifiers, while identifying stream data and novel-pattern use as future directions.
- PROSER mimics novel-class distributions with data placeholders, transforming closed-set training into open-set training.
- PROSER adds classifier placeholders that adaptively separate known from unknown classes using class-specific thresholds.
- PROSER generates novel classes with manifold mixup and adaptively sets the reserved open-set classifier value.
- Extending open-set recognition to stream data and using detected novel patterns remain future work.
I. Additional Experimental Results
This section presents additional experimental results and implementation details for the method.
- The section introduces additional experiment results.
- The section provides implementation details.
- The additional results precede the implementation details in this section.
I.1. Sensitivity about Hyper-Parameters
The sensitivity study evaluates loss-weight and dummy-classifier settings on CIFAR100 using macro F1. It finds that combining data and classifier placeholders performs best, while excessive dummy classifiers hurt recognition.
- Sensitivity about Hyper-Parameters: The study evaluates β, γ, and dummy classifier number C on CIFAR100 using macro F1 over 15 known classes and unknowns.Fifteen classes are known and another 15 are open-set categories, giving a known-unknown ratio of 1:1.
- Sensitivity about Hyper-Parameters: The ablations define Mixup as using only data placeholders and Dummy as using only dummy classifiers.
- Sensitivity about Hyper-Parameters: β = 1, γ = 0.1 leads to the best performance, indicating that data and classifier placeholders jointly improve the current task.
- Sensitivity about Hyper-Parameters: Multiple dummy classifiers increase placeholder diversity, but performance declines when C > 11.
I.2. Running Time Comparison
The running-time comparison on MNIST examines PROSER against softmax, OpenMax, and generative methods. PROSER is comparable to Softmax and OpenMax in order of magnitude and faster than the compared generative approaches.
- Running Time Comparison: The MNIST running-time comparison includes PROSER, OSRCI, GFROSR, OpenMax, and softmax, with the y-axis on a logarithmic scale.
- Running Time Comparison: PROSER has the same order of magnitude as Softmax and OpenMax.
- Running Time Comparison: OSRCI and GFROSR consume much more time than PROSER because they generate or reconstruct novel instances with additional generative models.
- Running Time Comparison: PROSER generates novel classes with manifold mixup without additional training steps for novel patterns.
I.3. Manifold Mixup VS Vanilla Mixup
The method generates data placeholders by mixing intermediate representations rather than inputs, and compares this approach with vanilla mixup. Manifold mixup produces a more compact embedding space and supports open-set recognition.
- Manifold mixup procedure: Manifold mixup computes pre-embeddings, shuffles embedding-label pairs, masks same-class pairs, and mixes the remaining pairs to form data placeholders.The mixed representations are passed through the post-embedding module and used in a manifold mixup loss.
- Manifold mixup procedure: The proposed procedure has no more total forward and backward consumption than vanilla training.Mixed embeddings are limited to the unmasked pairs, so their number does not exceed the mini-batch size.
- Mixup-weight distribution: α < 1 samples mixup weights near 0 or 1, α > 1 samples them near 0.5, and α = 1 yields a uniform distribution.The distribution of λ is governed by the Beta-distribution parameter α.
- Performance comparison: α = 2 achieves the best performance in the reported ablation, and the authors use α = 2 for all main-paper experiments without task-specific tuning.The same setting is retained across tasks rather than optimized separately for each one.
- Performance comparison: Manifold mixup generates instances more similarly to novel instances than vanilla mixup, whose generated space also overlaps the known space.The authors associate vanilla mixup's broader generated space with possible disruption of the learned embedding space.
II. Experiment Implementation
The experiments evaluate unknown detection with repeated-trial AUC reporting and fixed training settings across datasets. The CIFAR10 example indicates that crop outliers are easier to detect than resized full-image outliers.
- Evaluation protocol: Unknown detection performance is reported as AUC mean±std averaged over five randomized trials.Baseline results are taken from the cited prior works, and N.R. denotes an unavailable reported standard deviation.
- Training settings: PROSER uses the same backbone architecture as the referenced methods and fixes batch size to 128 across all datasets.Training uses SGD with momentum 0.9 and an initial learning rate of 0.001.
- Training settings: The experiments fix β = 1, γ = 0.1, five classifier placeholders, and α = 2 across datasets.The calibration bias is selected so that 95% of validation data are recognized as known.
- Dataset example: Crop outliers are easier to detect than resized full-image outliers in the CIFAR10 open-set example.The crop images contain only part of the original picture, whereas resize images preserve the full picture at 32*32 pixels; this agrees with the macro-F1 results.