Source-linked AI summary
Recent Advances in Open Set Recognition: A Survey
Chuanxing Geng, Sheng-jun Huang, Songcan Chen
TL;DR
Real-world recognition must handle test-time classes absent from training rather than assume shared closed-set label and feature spaces. This paper surveys OSR definitions, models, datasets, evaluation, related tasks, and open world recognition, while synthesizing limitations and research directions. Its review indicates differing method behavior across datasets and persistent challenges including adversarial samples and imperfect evaluation metrics.
Problem
Closed-set assumptions fail when unseen classes emerge during testing, motivating recognition methods that classify known classes while handling unknown classes.
Method
The paper comprehensively surveys OSR definitions, model categories, datasets, evaluation criteria, algorithm comparisons, related tasks, and open world recognition.
Results
Threshold-based methods can degrade substantially as openness increases, while CD-OSR achieves better performance on most datasets, especially LETTER and PENDIGITS.
Takeaways & Limitations
OSR research spans multiple modeling frameworks, but existing approaches still require analysis of limitations and promising directions for subsequent work.
Takeaways & Limitations
OSR evaluation can be distorted when known-class accuracy and unknown-class rejection are imbalanced, and adversarial samples remain a serious challenge.
Abstract
from arXiv · showhide
In real-world recognition/classification tasks, limited by various objective factors, it is usually difficult to collect training samples to exhaust all classes when training a recognizer or classifier. A more realistic scenario is open set recognition (OSR), where incomplete knowledge of the world exists at training time, and unknown classes can be submitted to an algorithm during testing, requiring the classifiers to not only accurately classify the seen classes, but also effectively deal with the unseen ones. This paper provides a comprehensive survey of existing open set recognition techniques covering various aspects ranging from related definitions, representations of models, datasets, evaluation criteria, and algorithm comparisons. Furthermore, we briefly analyze the relationships between OSR and its related tasks including zero-shot, one-shot (few-shot) recognition/learning techniques, classification with reject option, and so forth. Additionally, we also overview the open world recognition which can be seen as a natural extension of OSR. Importantly, we highlight the limitations of existing approaches and point out some promising subsequent research directions in this field.
1 INTRODUCTION
Open set recognition addresses non-stationary settings where unseen classes may appear at test time, requiring accurate known-class recognition alongside rejection or handling of unknown classes. The survey defines OSR’s relationships to related tasks and reviews models, evaluation, limitations, and research directions.
- Recognition categories: The survey distinguishes known known, known unknown, unknown known, and unknown unknown classes according to labeled samples and available side-information.Unknown unknown classes have neither training samples nor side-information, whereas unknown known classes have side-information available during training.
- OSR motivation: Closed-set methods can misclassify unknown samples because their decision boundaries allocate space to known classes.OSR instead limits known-class regions and reserves space for unknown samples.
- OSR motivation: Open set recognition assumes new classes unseen during training may appear during testing and requires classifiers to classify known classes while handling unknowns.Unknown samples require a corresponding reject option rather than forced assignment to a known class.
- Related tasks: OSR differs from reject-option classification because reject-option methods remain under the closed-set assumption and reject low-confidence samples from known classes.Zero-shot learning instead uses semantic information shared between known and unknown classes to recognize unknown known classes.
- Survey scope: The survey provides a comprehensive OSR review covering definitions, model representations, datasets, evaluation criteria, algorithm comparisons, related tasks, and open world recognition.It also discusses limitations of existing approaches and proposes subsequent research directions.
2 BASIC NOTATION AND RELATED DEFINITION
The section formalizes OSR through open space risk, openness, and open set risk. It explains how these concepts constrain recognition functions and motivates the survey’s recalibrated openness definition.
- Open space risk: Open space is the region far from known data, and labeling samples there as known classes incurs open space risk.The risk increases as more open-space samples are labeled as known classes.
- Openness: Openness measures how open a recognition problem is, with larger values indicating more open problems and zero indicating a completely closed problem.The section notes that the class-set relationships used in the original definition can produce unreasonable negative openness.
- Openness: The paper recalibrates openness so it depends on known-class knowledge from training and unknown-unknown knowledge from testing rather than all three class sets.The authors describe the revised form as more reasonable while acknowledging that other definitions may be more precise.
- Open set risk: Open set recognition seeks a measurable recognition function that minimizes open set risk over allowable recognition functions.The open set risk balances empirical risk with open space risk.
- Modeling implications: The formal concepts of open space risk and openness provide guidance for subsequent OSR modeling and algorithm development.The survey later categorizes approaches according to their modeling forms.
3 A CATEGORIZATION OF OSR TECHNIQUES
The survey categorizes OSR techniques by modeling perspective, covering traditional and deep discriminative methods alongside generative approaches, and maps their relationships. It also identifies open-space, adversarial, threshold-selection, and data-generation limitations that motivate future work.
- OSR modeling is organized into traditional ML and DNN methods from the discriminative perspective, plus instance- and non-instance-generation methods from the generative perspective.The survey reviews representative works in each category and links the approaches in a global overview.
- Closed-set assumptions cause traditional classifiers to over-occupy KKC regions, allowing UUC samples to fall inside regions assigned to known classes.OSR methods therefore seek constraints that limit open-space risk beyond the reasonable support of known classes.
- SVM-based OSR methods constrain decisions with slabs, compact abating probabilities, thresholding, or EVT-calibrated scores to reject likely UUC samples.The 1-vs-Set model adds a parallel hyperplane, while W-SVM uses two separated SVMs and Weibull calibration.
- Sparse-representation OSR can fail under extreme pose, illumination, or resolution variation and requires extensive training data spanning testing conditions.These limitations arise when the self-expressiveness property required by SRC no longer holds.
- OpenMax adapts deep networks by replacing SoftMax with an OpenMax layer, addressing fooling, rubbish, and unrelated open-set images.Its limitations include susceptibility to adversarial examples and a mismatch between the testing distance function and the training objective.
- Dirichlet-process methods adapt as data changes, while collective decision strategies can use testing-sample correlations and support new-class discovery.The survey highlights effective UUC generation, hierarchical Dirichlet processes, and collective decisions as promising directions.
4 BEYOND OPEN SET RECOGNITION
Open world recognition extends open set recognition by adding labeling and incremental learning for newly discovered unknown classes. It also faces stronger requirements, including scalability, continuous learning, and minimal downtime.
- Open set recognition is not incremental and does not scale gracefully with the number of classes.
- Open world recognition adds a labeling process that assigns labels to novel unknown data and identifies newly formed classes.
- Most existing OSR methods make decisions one sample at a time, even when samples arrive collectively in batches.
- Its incremental learning function adds new open-space-risk-managing recognition functions for the newly labeled classes.
- The NNO-based approach presumes supervised learning with labels currently obtained through human labeling, whereas ideally all OWR steps should be automated.
- OWR research additionally considers incremental metrics, confidence thresholds, local learning, minimal downtime, and continuous learning, but remains far from complete.
5.1 Datasets
OSR experiments commonly recast multiclass benchmarks by designating some labels as known classes for training and the remainder as unknown classes. The survey lists datasets spanning tabular, face, digit, natural-image, and larger-scale image settings.
- OSR benchmarks commonly select distinct labels as known classes and treat the remaining labels as unknown classes.
- LETTER contains 20,000 samples from 26 classes; 10 classes are used as known classes and the remainder as unknown classes.
- PENDIGITS contains 10,992 samples from 10 classes; 5 classes are used as known classes and the remainder as unknown classes.
- COIL20 and YALEB each use 10 randomly selected classes as known classes, with the remaining object or individual classes treated as unknown.
- MNIST, SVHN, and CIFAR10 use 6 known and 4 unknown classes, while CIFAR+10 and CIFAR+50 increase unknown-class openness.
- Tiny-Imagenet uses 20 known classes and 180 unknown classes from its 200-class dataset.
5.2 Evaluation Criteria
OSR evaluation must measure both known-class recognition and unknown-class rejection. The survey discusses accuracy extensions, F-measure, Youden’s index, AUROC, and normalized accuracy while noting unresolved metric limitations.
- OSR evaluation tracks true and false outcomes separately for known classes and unknown-class rejection.
- Accuracy for OSR: The OSR accuracy extension counts correct known-class classifications and correct unknown-class rejections, but this aggregate can misrepresent performance when class proportions are imbalanced.
- Accuracy for OSR: Normalized accuracy weights known-class accuracy AKS and unknown-class accuracy AUS using a regularization constant λr.
- F-measure: When applying F-measure, unknown classes should not be collapsed into one additional class because their correct classifications would incorrectly count as true positives.
- F-measure and Youden’s index: OSR F-measure calculations account for false unknown and known classes through false negatives and false positives, while Youden’s index incorporates true-negative performance.
- AUROC: AUROC is used with closed-set accuracy to reduce sensitivity to parameters and thresholds, but OSR ultimately requires a sample-level class-or-unknown decision.
- F-measure and AUROC are currently the most commonly used OSR metrics, while further evaluation methods remain worth exploring.
5.3 Experiments
The experiments compare representative OSR methods under non-depth and depth-feature settings using simulated closed- and open-set validation, with thresholds determined from known classes. Results show dataset- and openness-dependent performance, with CD-OSR generally strong on non-depth benchmarks and CROSR/C2AE strong on depth benchmarks, alongside important limitations.
- Evaluation Protocol: Thresholds are determined only from known known classes during training and remain fixed during testing because unknown unknown classes are unavailable beforehand.This recalibrates common openness-dependent threshold selection for the OSR setting.
- Evaluation Protocol: The evaluation separates fitting data containing known known classes from validation simulations representing closed- and open-set conditions.The protocol trains models on F and evaluates them on V, which contains both closed-set and open-set simulations.
- Non-depth Features: As openness increases, threshold-based methods can degrade significantly across datasets, whereas CD-OSR achieves better performance on most datasets, especially LETTER and PENDIGITS.The authors attribute CD-OSR’s advantage to HDP’s data-adaptation characteristic, which can model unknown unknown classes appearing during testing.
- Non-depth Features: OSNN shows large standard-deviation fluctuations, while 1-vs-Set and W-OSVM deteriorate as the setting becomes more open.The reported behavior is linked respectively to NNDR’s dependence on dataset distributions, unbounded open space, and one-class classification limitations.
- Limitations: The comparison is limited by model-specific weaknesses, EVT-related challenges with rare known and unknown classes, and protocols that may not fully characterize algorithm behavior.The paper also notes that CD-OSR can be difficult to apply to high-dimensional data and computationally complex.
- Depth Features: On depth-feature benchmarks, CROSR and C2AE generally outperform earlier methods, while MNIST results are comparable because performance is nearly saturated.The paper links CROSR’s results to joint classification and reconstruction, and C2AE’s to separating closed-set classification from open-set identification.
6 FUTURE RESEARCH DIRECTIONS
The paper identifies unresolved modeling and decision challenges in OSR, including compact representations for known classes, modeling unknown unknown classes, and interpreting rejected samples. It proposes hybrid learning, data generation, and new-class discovery as directions for further research.
- Modeling Perspectives: Existing OSR models are mainly discriminative or generative, motivating exploration of hybrid generative-discriminative models.The paper states that no OSR work had yet adopted this perspective at the time of discussion.
- Compact Class Regions: Clustering and classification could be unified to make target-class representations both more compact and more discriminative.Clustering is intended to constrain classes to tighter distribution areas, while classification supplies discriminativeness.
- Unknown Unknown Classes: Although unknown unknown classes cannot be modeled from known-class knowledge alone, adversarial generation and Dirichlet-process methods offer ways to approximate or account for them.The paper emphasizes that generating valid unknown-class data remains the key challenge for adversarial approaches.
- Decision and Rejection: Most OSR algorithms reject unknown samples but provide limited subsequent processing, motivating joint rejection, new-class discovery, and interpretability of the reject option.A rejected sample may correspond to a low-confidence target class, an outlier, or a new class.
6.3 About the Decision
The paper argues that OSR decisions should move beyond isolated sample rejection toward collective and information-enriched decisions. Correlations among test samples, semantic side-information, and auxiliary knowledge can support recognition or characterization of unknown classes.
- Collective Decision: Collective decisions can use correlations among testing samples and may enable new-class discovery, unlike predominantly individual-sample OSR decisions.The paper presents collective decision-making as a promising extension for batch or image-set inputs.
- Related Learning Settings: OSR can be combined with domain adaptation, semi-supervised learning, active learning, multi-task learning, multi-view learning, and multi-label classification.The paper notes that open-set settings have already been introduced into domain adaptation and semantic instance segmentation.
- Side-Information: Most existing OSR methods use only known-class feature information and leave semantic attributes, knowledge graphs, and known-unknown data underused.The paper identifies these forms of side-information as potentially important for improving performance.
- Side-Information: Shared semantic or attribute information between known classes and unknown data could help characterize unknown samples instead of simply rejecting them.This proposed use differs from ZSL and generalized ZSL, which assume semantic information for unknown classes during training.
- Side-Information: Known-unknown data such as universum samples can shrink known-class regions and thereby reduce open-space risk.The paper illustrates this direction using auxiliary data outside the classes of interest.
6.6 Relative Open Set Recognition
Relative open set recognition addresses settings that are closed at a coarse level but open when finer-grained classes must be identified. The paper also suggests integrating classifiers trained on separate sub-knowledge sets to reduce open-space risk.
- Relative Open Set Recognition: Medical diagnosis can be closed for sick-versus-healthy detection yet become a complete OSR problem when unseen disease types appear.The relevant goal is to limit the scope of unknown classes while finding the most specific label supported by the known-class taxonomy.
- Multiple Knowledge Sets: Combining classifiers trained on different sub-knowledge sets is proposed as a way to further reduce open-space risk.The paper frames this integration as an interesting but challenging future-work topic.
7 CONCLUSION
Because real-world recognition cannot model everything, open set recognition remains widespread and challenging. This survey comprehensively reviews OSR techniques and related tasks while identifying limitations and future research directions.
- Real-world recognition is constrained by the impossibility of modeling everything, making the OSR scenario ubiquitous.
- The survey covers OSR definitions, model representations, datasets, evaluation criteria, and algorithm comparisons.
- It analyzes relationships with related tasks, reviews open world recognition, discusses existing limitations, and proposes subsequent research directions.