Source-linked AI summary
Generative OpenMax for Multi-Class Open Set Classification
ZongYuan Ge, Sergey Demyanov, Zetao Chen, Rahil Garnavi
TL;DR
Multi-class open set classification must identify unknown classes rather than misclassify them as known, while estimating known-class probabilities accurately. G-OpenMax extends OpenMax with GAN-generated synthetic unknown samples for explicit unknown-class probability estimation and visualization. Results are better when its shared-feature assumption holds, though natural-image performance remains limited.
Problem
Multi-class open set classification requires simultaneously estimating known-class probabilities and recognizing classes absent from training.
Method
G-OpenMax extends OpenMax by generating synthetic unknown samples from mixtures of known-class latent distributions and using them as an additional training label.
Results
Results demonstrate better performance when the assumption that classes share common features holds, with evaluation on 10-class and 95-class openness problems.
Takeaways & Limitations
G-OpenMax provides explicit probability estimation and visualization of unknown classes, rather than inferring unknowns only from known-class scores or distances.
Takeaways & Limitations
On ImageNet, G-OpenMax shows no obvious improvement over the natural-image setting because generated images may not plausibly represent unknown classes.
Abstract
from arXiv · showhide
We present a conceptually new and flexible method for multi-class open set classification. Unlike previous methods where unknown classes are inferred with respect to the feature or decision distance to the known classes, our approach is able to provide explicit modelling and decision score for unknown classes. The proposed method, called Gener- ative OpenMax (G-OpenMax), extends OpenMax by employing generative adversarial networks (GANs) for novel category image synthesis. We validate the proposed method on two datasets of handwritten digits and characters, resulting in superior results over previous deep learning based method OpenMax Moreover, G-OpenMax provides a way to visualize samples representing the unknown classes from open space. Our simple and effective approach could serve as a new direction to tackle the challenging multi-class open set classification problem.
1 Introduction
Multi-class open set classification must recognize both known classes and previously unseen classes, balancing specialization in known space with generalization to unknown open space. G-OpenMax addresses this by explicitly modelling unknown categories with GAN-generated samples and evaluates the approach across small- and large-scale openness settings.
- Open set classification requires rejecting test objects outside the known classes instead of falsely assigning them to known categories.
- Multi-class open set classification jointly requires accurate probabilities for all known classes and precise predictions for unknown classes.
- Recent methods estimate unknown probabilities through score calibration and statistical modelling, but model open-space risk mainly in decision space rather than pixel space.
- G-OpenMax extends OpenMax by using GANs to generate synthetic unknown-class samples and provide explicit probability estimation over unknown categories.
- G-OpenMax enables visualization over known and unknown classes while evaluating effectiveness and limitations on 10-class and 95-class openness problems.
2 Background
Open set recognition developed from adapted closed-set classifiers toward methods that estimate or detect unknown classes. OpenMax combines deep networks with calibrated scores and meta-recognition, while GANs provide a generative basis for image synthesis.
- Early open set methods adapted binary or multi-class classifiers to detect unknowns using decision-space distances and compact abating probability models.
- Figure 1 distinguishes known space, containing trained classes, from unknown open space, containing classes absent during training.
- Open world recognition introduced active detection and learning of new classes through the Nearest Non-Outlier algorithm.
- OpenMax adapts deep learning with penultimate-layer representations and meta-recognition to reject fooling or unknown open-set classes.
- GANs learn complex data distributions through competition between generator and discriminator networks and can synthesize plausible grayscale and RGB images.
3 Methodology
G-OpenMax extends OpenMax by generating synthetic unknown-class samples and training an explicit unknown output alongside known classes. Its pipeline combines GAN-based sample generation and selection with a two-stage classifier and Weibull score calibration.
- G-OpenMax directly estimates unknown-class probability by adding GAN-generated synthetic images as an extra training label.This contrasts with OpenMax's aggregation of calibrated known-class scores to form a pseudo unknown probability.
- G-OpenMax trains NetG with K + 1 classes, then uses its extra activation dimension for unknown samples before applying OpenMax-style Weibull calibration.The additional dimension supplies feature-space information relating known-class mean activation vectors to the unknown-class mean.
- The method assumes unknown classes lie in a subspace of the original domain rather than coming from arbitrary distributions.This assumption excludes test objects from other datasets and motivates generating domain-related unknown samples.
- Synthetic unknown samples are produced by mixing known-class distributions in latent space, where interpolation is intended to preserve plausible class-domain structure.The class subspace is lower-dimensional than pixel space, so generated open-set images occupy a small nonlinear subspace.
- The implementation uses a modified conditional GAN, with generator and discriminator architectures specified in separate layer tables.The GAN is optimized with Adam using learning rate 2e-4 and β1 = 0.5, β2 = 0.999, ε = 10^-8.
- Generated samples are screened with the pretrained classifier, and incorrectly predicted samples are selected as candidates for the unknown class.This selection step targets samples distinct from known classes while remaining representative of the assumed open space.
4 Experiments
Experiments compare G-OpenMax and related methods on MNIST and HASYv2 under varying openness, tail sizes, and thresholds. Results show consistent gains from GAN-based synthetic samples, while sample quality limits performance on more complex data and natural images.
- Experimental Setup: Experiments use MNIST and HASYv2, with 60% training, 20% validation, and 20% testing samples under three-fold cross-validation.HASYv2 experiments retain classes with at least 500 samples; 20 training classes are sampled per generator iteration because training on 60 classes is difficult to converge.
- Experimental Setup: The openness value varies the number of known training classes against the total classes recognized during testing.The MNIST setup holds out six classes for training and varies openness over the remaining four; HASYv2 uses 60 training and 35 unknown classes.
- Main Results: G-OpenMax provides consistent improvement across MNIST and HASYv2, while SoftMax F-measure degrades rapidly as openness increases.G-SoftMax is competitive with OpenMax on MNIST and tolerates increasing openness better than normal SoftMax, whereas poorer HASYv2 mixture quality hinders both GAN-based methods.
- Ablation Experiments: G-OpenMax maintains overall performance with extremely large Weibull tail sizes, despite large tails increasing rejection of both unknown and known samples.The figure reports that G-OpenMax consistently outperforms OpenMax on known-sample recognition under large tail sizes.
- Ablation Experiments: GAN-trained methods consistently outperform methods without GAN samples on unknown test samples, with G-OpenMax nearly 10% better than OpenMax at the optimal threshold.On MNIST, unknown-class accuracy continues improving up to tail size 60, whereas large tail sizes perform poorly on HASYv2.
- Qualitative Analysis: On ImageNet12, generated mixtures can blend into unrealistic objects or collapse toward a single class appearance, yielding no obvious G-OpenMax improvement over OpenMax.The paper relates this limitation to the variety of ImageNet classes and the lack of common features in the learned latent space.
- Qualitative Analysis: Generated MNIST mixtures produce digit-like unknown samples such as 8 and 9, while HASYv2 mixtures can contain random dots and be difficult for humans to interpret.The paper attributes the HASYv2 difficulty to its complexity and variation across English alphabets, Greek letters, and other character classes.
5 Conclusion
The paper concludes that G-OpenMax extends OpenMax with GAN-based unknown-class modelling and visualization. Its results are better when the assumption that classes share common features holds, while natural-image open-set classification remains unresolved.
- Conclusion: G-OpenMax extends OpenMax with GANs to provide explicit probability estimation and visualization for unknown classes.The method relies on a reasonable open-space modelling assumption that all classes share common features.
- Conclusion: Results are better when the assumption that all classes share common features holds.The conclusion identifies mixture-class complexity and natural-image open-set classification as unanswered questions for future work.