Source-linked AI summary
C2AE: Class Conditioned Auto-Encoder for Open-set Recognition
Poojan Oza, Vishal M Patel
TL;DR
Open-set recognition must identify unknown classes during testing while preserving known-class classification, but deep-neural-network scores and operating thresholds remain relatively unexplored. This paper uses class-conditioned auto-encoders with separate classification and identification stages, applying EVT to reconstruction errors; experiments across image-classification datasets report significant improvements over state-of-the-art methods.
Problem
Open-set recognition addresses identifying unknown test samples while correctly classifying known classes, a challenge because training provides access only to known classes.
Method
Class-conditioned auto-encoders separate closed-set classification from open-set identification, using conditional reconstruction and Extreme Value Theory to set operating thresholds.
Results
The proposed approach performs well across multiple image-classification datasets and reports significant improvements over many state-of-the-art open-set algorithms.
Takeaways & Limitations
Separating open-set recognition into subtasks helps learn better open-set identification scores and supports finding an operating threshold.
Takeaways & Limitations
The method assumes the prior probability of unknown samples is a function of openness, using pu = 0.5 ∗O in one experiment.
Abstract
from arXiv · showhide
Models trained for classification often assume that all testing classes are known while training. As a result, when presented with an unknown class during testing, such closed-set assumption forces the model to classify it as one of the known classes. However, in a real world scenario, classification models are likely to encounter such examples. Hence, identifying those examples as unknown becomes critical to model performance. A potential solution to overcome this problem lies in a class of learning problems known as open-set recognition. It refers to the problem of identifying the unknown classes during testing, while maintaining performance on the known classes. In this paper, we propose an open-set recognition algorithm using class conditioned auto-encoders with novel training and testing methodology. In contrast to previous methods, training procedure is divided in two sub-tasks, 1. closed-set classification and, 2. open-set identification (i.e. identifying a class as known or unknown). Encoder learns the first task following the closed-set classification training pipeline, whereas decoder learns the second task by reconstructing conditioned on class identity. Furthermore, we model reconstruction errors using the Extreme Value Theory of statistical modeling to find the threshold for identifying known/unknown class samples. Experiments performed on multiple image classification datasets show proposed method performs significantly better than state of the art.
1. Introduction
Open-set recognition addresses the need to classify known classes correctly while identifying samples from unseen classes as unknown. The paper proposes separating classification from open-set identification and using class-conditioned auto-encoders with thresholding based on extreme value modeling.
- Problem Setting: Open-set recognition jointly classifies samples from the known set and identifies samples from the unknown set as unknown.Known examples must be assigned to their classes, while unknown examples must be rejected as unknown.
- Motivation: Closed-set classifiers can force samples from classes unseen during training into one of the known classes.This creates a need to identify test samples as known or unknown while preserving known-class classification.
- Challenge: Unknown-sample identification is difficult because training provides incomplete knowledge of the world and only known classes are accessible.
- Related Challenge: Recognition-score thresholding raises questions about which score to use and how to select its operating threshold, especially for deep neural networks.These questions have been explored more extensively for traditional models than for deep neural networks.
- Proposed Approach: The proposed method divides training into closed-set classification and open-set identification, trained separately in a stage-wise manner.The authors report that this strategy provides good identification scores and enables selection of an operating threshold.
- Contributions: The paper introduces class-conditioned auto-encoder training and testing, evaluates it on multiple image-classification datasets, and compares it with recent state-of-the-art methods.Ablation experiments analyze the effectiveness of the proposed method.
2. Related Work
Prior open-set recognition work includes traditional score-calibration methods and neural approaches that modify recognition scores or generate unknown-like data. The proposed method instead uses class-conditioned auto-encoders and models reconstruction-error extremes for threshold selection.
- Traditional Methods: Traditional open-set methods extend classifiers such as SVMs, nearest neighbors, and sparse representation with score-based recognition strategies.Extreme value distributions have been used to calibrate decision scores in traditional methods.
- Neural Methods: Neural approaches include OpenMax activation-vector calibration, GAN-based unknown-sample generation, counterfactual image generation, and k-sigmoid losses.These methods adapt closed-set networks or their scores for open-set identification.
- Proposed Method: The proposed method separates closed-set classification from open-set identification rather than requiring one score to serve both purposes.Its training procedure uses class-conditioned auto-encoders in separate stages.
- Threshold Modeling: Extreme value theory models the tails of recognition-score distributions where match and non-match scores overlap.The paper applies this statistical framework to reconstruction errors for operating-threshold estimation.
3. Proposed Method
The proposed method separates open-set recognition into closed-set classification and open-set identification, training an encoder-classifier and a class-conditioned decoder for distinct roles. Reconstruction-error distributions are modeled with extreme value theory to estimate an operating threshold, and inference evaluates all class conditions.
- Stage 1: Closed-set Training: The encoder F and classifier C are trained on labeled batches with cross-entropy loss for k known classes.Each batch contains N images and labels y_i ∈ {1, 2, ..., k}; p_yi is the predicted probability vector.
- Stage 2: Conditional Decoder Training: The decoder is conditioned through FiLM so matching labels produce accurate reconstructions while non-matching labels produce poor reconstructions.Non-matching examples are sampled with class identities inconsistent with the input, encouraging reconstruction errors that distinguish conditions.
- Inference: The k-inference algorithm conditions the decoder on every possible class label and obtains k reconstruction errors for each test sample.These errors support identifying whether a sample matches a known class condition or remains poorly reconstructed under all conditions.
- EVT Modeling: Extreme value theory models the tails of match and non-match reconstruction-error distributions with generalized Pareto distributions.The right tail of match errors and left tail of non-match errors are modeled; the latter is inverse-transformed because GPD models maxima.
- Threshold Calculation: The operating threshold τ* is searched over the overlap of match and non-match errors to minimize threshold-dependent error probability.Its estimation depends on how well the training-derived error sets represent known and unknown distributions, and on the unknown-sample prior p_u.
4. Experiments and Results
Experiments evaluate open-set identification and overall recognition across several image datasets, comparing the proposed method with established baselines and ablations. The proposed approach outperforms competing methods, remains stable as openness increases on LFW, and benefits from non-match reconstruction loss, EVT thresholding, and openness-dependent priors.
- Evaluation protocol: The evaluation measures open-set identification with AUROC and overall open-set recognition with F-measure across multiple image classification datasets.AUROC is averaged over five randomized trials, while LFW is used for F-measure evaluation under varying openness.
- Datasets and openness: The experiments vary openness across MNIST, SVHN, CIFAR10, CIFAR+10, CIFAR+50, TinyImageNet, and LFW protocols.For MNIST, SVHN, and CIFAR10, openness is O = 13.39%; CIFAR+10 and CIFAR+50 use O = 33.33% and 62.86%, respectively.
- Comparison with state-of-the-art: The proposed method outperforms comparison methods for open-set identification, with minor gains on digits and significantly better performance on CIFAR and TinyImageNet.The comparison includes SoftMax, OpenMax, G-OpenMax, and OSRCI.
- Comparison with state-of-the-art: On LFW, the proposed approach remains relatively stable as openness increases and outperforms W-SVM, SROR, and DOC.DOC performs better initially, but its performance suffers significantly beyond 50% openness, whereas W-SVM and SROR remain relatively stable.
- Ablation Study: The CLS SoftMax-thresholding baseline has the weakest ablation performance on LFW.This baseline thresholds SoftMax probabilities from a closed-set model.
- Ablation Study: The Naive baseline remains relatively stable with increasing openness, showing the importance of the non-match reconstruction loss L_nm^r for threshold selection.The non-match loss avoids relying on heuristic thresholding of reconstruction scores.
- Ablation Study: EVT modeling of reconstruction errors improves thresholding over raw score values, while setting p_u = 0.5 ∗ O yields a marginal improvement over fixed p_u = 0.5.At large openness values, the fixed-prior and openness-dependent-prior variants achieve similar performance.
5. Conclusion
The paper concludes that class conditioned auto-encoders and separate training sub-tasks improve open-set identification across image classification datasets. The method estimates operating thresholds from reconstruction-error distributions but requires k feed-forwards per test sample, increasing test time.
- Conclusion: The proposed algorithm uses class conditioned auto-encoders with separate training and testing strategies for open-set recognition.The approach divides open-set recognition into closed-set classification and open-set identification sub-tasks.
- Conclusion: Separating open-set recognition into sub-tasks helps learn a better score for identifying known and unknown samples.Conditional reconstruction constraints help approximate known and unknown reconstruction-error distributions for threshold calculation.
- Conclusion: The proposed approach performs well across multiple image classification datasets and provides significant improvements over many state-of-the-art open-set algorithms.
- Conclusion: Inference for a single sample requires k feed-forwards, resulting in increased test time.The conclusion identifies this computational cost as a limitation of the proposed approach.
6. Supplementary Material for C2AE: Class Conditioned Auto-Encoder for Open-set Recognition
The supplementary material provides toy-data decision-boundary analyses, reconstruction-error progression, AUROC reporting context, and the architecture used for LFW experiments.
- Supplementary material: The supplement describes additional details for the proposed C2AE method.These details are provided because of space limitations in the submitted paper.
- Toy examples: Toy experiments use Two-Gauss, Four-Gauss, and Uni-Gauss 2D datasets to examine learned decision boundaries.The encoder, decoder, and classifier use fully connected sigmoid networks.
- Toy examples: The proposed approach learns tight boundaries around toy data points and identifies the remaining space as unknown.This behavior is reported for all three 2D dataset variations.
- Histogram progression: SVHN reconstruction-error histograms compare match and non-match data at iterations 1, 5k, and 500k.Initially the errors are not discriminative, but they become more discriminative as training proceeds.
- Histogram progression: Known and unknown reconstruction errors follow the same training trend as match and non-match errors on SVHN.The histograms are normalized for match, non-match, known, and unknown samples.
- LFW architecture: The LFW experiments use a U-Net-inspired architecture with a FiLM conditioning layer in the middle.The architecture combines convolutional and transposed-convolutional layers with instance normalization and leaky ReLU activations.