Learning a Neural-network-based Representation for Open Set Recognition

1 INTRODUCTION

• Open set recognition must handle novel or unknown categories at test time rather than force every instance into a known class.The introduction illustrates the issue with a cat-and-dog classifier faced with a fox or caracal.Source: §1 · p. 1
• Face recognition is another cited application, where a system must determine whether a query face matches someone in a known gallery.Ortiz and Becker describe automatic friend labeling in social media posts.Source: §1 · p. 1
• Novel malware families emerge regularly, leaving malware training data incomplete and creating an open-set classification scenario.Source: §1 · p. 1
• The paper proposes a neural-network-based representation and a mechanism for open set recognition, evaluating the approach on malware and images.Source: §1 · p. 1
• Statistically significant improvements over previous research work on three datasets are reported for the proposed approach.The evaluation spans two malware datasets and images.Source: §1 · p. 1

2 RELATED WORK

• One class of systems identifies known versus unknown instances but does not distinguish among multiple known classes.Examples include 1-vs-set binary SVM and KNFST, with extending KNFST using sample locality for outlier scoring.Source: §2 · p. 1
• A second class both identifies unknown instances and discriminates among known classes.PI-SVM uses one binary SVM per class and fits a Weibull distribution to each classifier's scores; NCM extensions add incremental learning.Source: §2 · p. 1
• Openmax uses class-mean activation-vector distances for test-time unknown-class handling, although standard cross-entropy training does not directly encourage instances to cluster around those means.Source: §2 · p. 2
• Ge et al. [12] combine Openmax with GAN-generated synthetic known-class instances, while other cited work addresses malware clustering or open-set intrusion detection.The malware clustering work was not proposed for open set recognition, though its outlier score can be used for unsupervised open set recognition.Source: §2 · p. 2

3 APPROACH

• Given labeled training instances from K known classes, open set recognition learns to assign an unseen instance to a known class or an unknown class.Source: §3 · p. 2
• Unlike closed-set classification, the learned function must handle unseen instances that may belong to classes not known during training.Source: §3 · p. 2
• A further challenge is learning an instance representation that facilitates open set recognition better than the original representation.Source: §3 · p. 2

3.1 Overview

• The learned representation resembles a Fisher discriminant in separating classes, but uses a neural network's nonlinear projection.Source: §3.1 · p. 2
• A loss function directly encourages hidden-layer representations to bring same-class instances closer and different-class instances farther apart.The approach targets properties that ordinary closed-set classification losses such as cross-entropy may not produce.Source: §3.1 · p. 2
• Distance from a class center provides an outlier score, with greater distance indicating greater likelihood of an unknown-class instance.A threshold is estimated from training-instance distances to identify unknown instances.Source: §3.1 · p. 2
• The final prediction first uses the outlier-score threshold to separate known from unknown instances, then assigns known instances their most likely known-class label.Source: §3.1 · p. 2

3.2 Learning representations

• The final linear-layer output is treated as the input instance's projection into a new representation space, learned through a function mapping x to z.Source: §3.2.1 · p. 2, §3.2 · p. 2

3.2.1 II-Loss Function.

• Inter-class separation is measured by the distance between the closest pair of class means, avoiding an average-distance alternative that could be dominated by the largest separation.The alternative is described as not resulting in good separation.Source: §3.2.1 · p. 3
• ii-loss minimizes intra-class spread and maximizes inter-class separation by training the network with mini-batch stochastic gradient descent and backpropagation.After training, class means are calculated from all training instances and stored with the model.Source: §3.2.1 · p. 3

3.2.2 Combining ii-loss with Cross Entropy Loss.

• Combining ii-loss with cross-entropy adds a softmax classifier to target low training classification error alongside representation separation.The network updates its weights first to minimize ii-loss and then to minimize cross-entropy.Source: §3.2.2 · p. 3

3.3 Outlier Score for Open Set Recognition

• The outlier score is the distance from a test instance's projection to the nearest known-class mean; greater distance makes it more likely to be an outlier.Source: §3.3 · p. 3
• The threshold is the training-set outlier-score percentile implied by an assumed contamination ratio; for example, a 0.01 ratio uses the 99th percentile.Source: §3.3.1 · p. 3

3.3.1 Threshold Estimation.

• A global outlier threshold consistently gives more accurate results than a separate threshold for each class.Source: §3.3.1 · p. 3

• The known class assigned to a test instance is the one whose class mean is nearest to its projection.Source: §3.4 · p. 4
• A probability distribution over known classes is obtained by applying softmax to the negative distances between the test projection and all known-class means.Source: §3.4 · p. 4

• The classifier assigns label K +1 when the outlier score exceeds the threshold; otherwise, it selects a known-class label.Source: §3.5 · p. 4
• For the known-class decision, probabilities come from Equation 6 with ii-loss alone, or from the softmax layer when training combines ii-loss and cross-entropy.Source: §3.5 · p. 4

4.1 Datasets

• The evaluation uses Microsoft malware, Android malware, and MNIST; the Android set has 986 samples after classes with fewer than 40 samples are removed.Microsoft malware features are function call graphs reformatted as adjacency matrices for convolutional layers.Source: §4.1 · p. 4

4.2 Simulating Open Set Dataset

• Open-set datasets are simulated by randomly designating classes as known or unknown, then withholding unknown-class instances from training.Three random open-set datasets are created per source dataset; Microsoft and Android use six known and three unknown classes, while MNIST uses six known and four unknown.Source: §4.2 · pp. 4–5

4.4 Detecting Unknown Class Instances

• ii and ii+ce significantly outperform the cross-entropy baseline in unknown-instance detection by AUC up to both 100% and 10% FPR.The AUC comparison excludes Openmax because it does not provide a specific outlier score; the table reports 30-run averages.Source: Table 1 · p. 5, §4.4 · p. 5

4.3 Evaluated Approaches

• The four evaluated systems are ii, ii+ce, cross-entropy-only ce, and reimplemented Openmax; the malware and MNIST networks use convolutional layers.The Android network uses fully connected layers only; the reported Openmax configuration combines Euclidean and cosine distance.Source: §4.3 · p. 5

4.5 Open Set Recognition

• On all three datasets, ii and ii+ce achieve significantly better open-set F-scores than ce and Openmax (p ≤ 0.0002).Scores average precision, recall, and F-score over K known classes and one unknown class, across 30 runs.Source: §4.5 · p. 5, Figure 2 · p. 6

4.6 Closed Set Classification

• Closed-set ii-loss performance is comparable to cross-entropy on MNIST and Android, and ii+ce is comparable to cross-entropy on MNIST.MNIST accuracies are 99.42% for cross-entropy, 99.31% for ii-loss, and 99.40% for ii+ce; Android accuracies are 93.10% for cross-entropy and 92.68% for ii-loss (p = 0.43).Source: §4.6 · p. 6

4.7 Discussions

• ii-loss projects known instances near their class regions and unknowns mostly between them, supporting class-mean-distance outlier scoring; Openmax projections concentrate along axes.The z-layer dimension was set equal to the number of classes after tuning; batch normalization bounds its output and prevents the inter-separation term from increasing indefinitely.Source: §4.7 · p. 7
• Openmax recall is low for Android classes with few training instances, while cross-entropy can project unknowns near known classes, lowering known-class precision.The authors attribute lower performance across networks on Android to its small, imbalanced dataset, including classes with fewer than 60 samples.Source: §4.5 · p. 6, §4.7 · p. 7

• The representation brings same-class instances closer and different-class instances farther apart, creating larger spaces for unknown instances to occupy.Source: §5 · p. 7
• Across malware and image datasets, the approach achieves statistically significant improvement over a cross-entropy baseline and Openmax.Source: §5 · p. 7
• The proposed threshold estimator is simple; exploring a more robust estimation method remains future work.Source: §5 · p. 7

• For MNIST, the networks use padded 32×32 inputs, convolutional and fully connected layers, and a six-dimensional linear z-layer.The architecture description also specifies batch normalization, dropout, and Adam training settings.Source: A EVALUATION NETWORK ARCHITECTURES · pp. 7–8
• Figures 3–7 depict MNIST known and unknown z-layer projections, Android Openmax and cross-entropy projections, inter-class separation with and without z-layer batch normalization, and autoencoder hidden-layer projections.Source: Figure 3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
• The MS Challenge network uses padded 67×67 inputs, one 256-unit fully connected layer, and dropout with keep probability 0.9.It is trained with Adam using a 0.001 learning rate, 0.9 beta1, and 0.999 beta2.Source: A EVALUATION NETWORK ARCHITECTURES · pp. 8–9
• The Android network uses a 64-unit fully connected layer followed by a six-dimensional z-layer, with batch normalization and dropout.It is trained with Adam for 10,000 iterations; the closed-set experiments use z dimensions of 10 for MNIST and 9 for MS and Android.Source: A EVALUATION NETWORK ARCHITECTURES · p. 9