Source-linked AI summary
Towards Open Set Deep Networks
Abhijit Bendale, Terrance Boult
TL;DR
Deep networks need to reject unknown inputs because closed-set classification can assign high confidence to meaningless or unseen images. The paper introduces OpenMax, which adapts Meta-Recognition to penultimate-layer activation patterns to estimate unknown probability. OpenMax rejects many unknown and fooling images, provides a formal open-set recognition model, and has only modest impact on true classification rate.
Problem
Closed-set deep networks must choose among known classes, while SoftMax thresholding fails to reliably reject visually meaningless or unknown inputs.
Method
OpenMax extends SoftMax by using Meta-Recognition on penultimate-layer activation vectors to estimate whether inputs are far from known training data and assign unknown probability.
Results
OpenMax rejects many unknown open-set and fooling images, rejects some adversarial images, and has only modest impact on true classification rate.
Takeaways & Limitations
OpenMax can serve as a novelty detector for incremental learning or flag rejected inputs for other modalities.
Takeaways & Limitations
OpenMax can fail on adversarial images targeting nearby fine-grained classes, and adversarial-image evaluation lacks a meaningful generation distribution.
Abstract
from arXiv · showhide
Deep networks have produced significant gains for various visual recognition problems, leading to high impact academic and commercial applications. Recent work in deep networks highlighted that it is easy to generate images that humans would never classify as a particular object class, yet networks classify such images high confidence as that given class - deep network are easily fooled with images humans do not consider meaningful. The closed set nature of deep networks forces them to choose from one of the known classes leading to such artifacts. Recognition in the real world is open set, i.e. the recognition system should reject unknown/unseen classes at test time. We present a methodology to adapt deep networks for open set recognition, by introducing a new model layer, OpenMax, which estimates the probability of an input being from an unknown class. A key element of estimating the unknown probability is adapting Meta-Recognition concepts to the activation patterns in the penultimate layer of the network. OpenMax allows rejection of "fooling" and unrelated open set images presented to the system; OpenMax greatly reduces the number of obvious errors made by a deep network. We prove that the OpenMax concept provides bounded open space risk, thereby formally providing an open set recognition solution. We evaluate the resulting open set deep networks using pre-trained networks from the Caffe Model-zoo on ImageNet 2012 validation data, and thousands of fooling and open set images. The proposed OpenMax model significantly outperforms open set recognition accuracy of basic deep networks as well as deep networks with thresholding of SoftMax probabilities.
1 Introduction
Deep networks excel in closed-set recognition but can assign high confidence to meaningless or unknown images. The paper introduces OpenMax, using penultimate-layer activation patterns and Meta-Recognition to estimate unknown probability and support open-set rejection.
- Deep networks assume all categories are known in advance, creating operational problems in dynamic real-world settings.
- Unknown objects cannot be exhaustively represented by an “other” training class, motivating methods for unknown unknowns.
- High SoftMax confidence on visually meaningless fooling or rubbish images shows that uncertainty thresholding alone cannot reliably identify unknown inputs.
- OpenMax uses penultimate-layer activation vectors and Meta-Recognition to estimate whether an input is far from known training data and assign unknown probability.
- Measuring open space risk in feature space enables rejection probabilities for unknown, fooling, and many adversarial images.
- The paper formalizes open-set deep networks, proves bounded open space risk, and evaluates rejection of unknown classes, fooling images, and obvious adversarial errors while maintaining test accuracy.
2 Open Set Deep Networks
OpenMax adapts deep networks for open set recognition by estimating unknown-class probability from penultimate-layer activation patterns. It combines distance-based meta-recognition with revised SoftMax probabilities to reject unknown inputs while formally limiting open space risk.
- Thresholding SoftMax uncertainty helps reject some unknown inputs but remains a relatively weak open set recognition tool.
- OpenMax replaces SoftMax with a model that estimates unknown-class probability from penultimate-layer activation vectors.It uses meta-recognition models and distances from class-specific learned means.
- Each class is represented by a mean activation vector computed from correctly classified training examples, and test inputs are assessed by their distance from these means.
- Open set inputs can lie far from a class mean, but related or poorly distinguished unknown objects may remain close, so uncertainty thresholding is also retained.
- OpenMax can reject open set, fooling, and some adversarial images by recalibrating activation scores and adding an explicit unknown class.Adversarial images targeting nearby fine-grained classes may evade detection because their activation patterns remain close.
- The OpenMax transformation forms a compact abating probability model, providing a formal open set recognition function with bounded open space risk.
3 Experimental Analysis
The evaluation tests OpenMax on known, fooling, and previously unseen ImageNet categories, using penultimate-layer activation statistics to calibrate unknown-class probabilities. OpenMax is compared with SoftMax thresholding for open set detection and classification.
- OpenMax Calibration: OpenMax computes class-specific mean activation vectors from correctly classified FC8 training examples and uses distances to those vectors for calibration.Mean activation vectors are computed separately for each class and crop/channel.
- Evaluation Setup: The test protocol combines 50,000 ILSVRC 2012 validation images, 15,000 open set images from unused ILSVRC 2010 categories, and 15,000 fooling images.The evaluation uses 80,000 images in total.
- Results: Nearly 4.3% higher accuracy than SoftMax with an optimal threshold and 12.3% higher accuracy than the base deep network are reported for OpenMax.Over the test set, OpenMax correctly classified 3,450 more images than SoftMax and 9,847 more than the base network.
- OpenMax Calibration: During testing, FC8 activations are compared with class-specific mean activation vectors and Weibull parameters to produce revised activations, including an unknown class.OpenMax probabilities are then computed from the revised activations for each channel.
- Evaluation Metrics: F-measure evaluates OpenMax and SoftMax across probability thresholds, counting correct known classifications and known-class errors on fooling and open set images.False negatives include fooling and open set images incorrectly classified as known examples.
- Results: OpenMax performs well as a detector for fooling images and open set test images across varying probability thresholds.The figure specifically reports strong rejection performance for fooling images during prediction.
4 Discussion
The discussion describes OpenMax as a rejection mechanism for unknown, fooling, and some adversarial inputs, while also identifying uses for rejected or misclassified images. It notes parameter trade-offs and illustrates how cropping or preprocessing can recover class evidence.
- Rejection and Applications: OpenMax automatically rejects many unknown, fooling, and some adversarial images while having only modest impact on the true classification rate.Rejected inputs could be routed to human labeling, incremental learning, or other modalities.
- Rejection and Applications: Rejected inputs can support novelty detection, human labeling, incremental learning of new categories, or escalation to other modalities.These uses are presented as operational possibilities rather than fixed components of OpenMax.
- Failure Analysis: In the agama example, cropping separates jeep and agama regions, producing accepted probabilities of 0.32 and 0.21 after the full image rejects the agama mean activation vector.The full image’s highest-scoring class is jeep with probability 0.26.
- Failure Analysis: A small amount of Gaussian blur changes an adversarial hammerhead image from rejection to acceptance as hammerhead shark with probability 0.79.The base deep network had incorrectly classified the noisy image as a scuba diver.
- Limitations and Trade-offs: Increasing open set and fooling rejection can reject more true classes, while multiple mean activation vectors may better capture contextual variation within a class.The paper gives baseball-on-a-desk and baseball-thrown-by-a-pitcher as contrasting contexts.
- Failure Analysis: OpenMax rejection often identifies incorrectly classified ImageNet images, especially images containing multiple objects.The discussion suggests this may aid training and localization development.
5 Towards Open Set Deep Networks: Supplemental
The supplement adds experiments examining OpenMax parameter behavior and sensitivity to the number of top classes used in the algorithm.
- Supplemental Experiments: Supplemental experiments examine performance across parameters for EVT calibration and sensitivity to the number of top classes, α, in Algorithm 2.These experiments use the ILSVRC 2012 dataset.
6 Parameters for OpenMax Calibration
OpenMax calibration balances rejection of open-set and fooling images against retaining correct validation images. The experiments identify tail size 20, α = 10, and euclidean-cosine distance as effective choices.
- Tail Sizes for EVT Calibration: Tail sizes beyond 20 improve rejection of open-set and fooling images but reduce validation-set performance, so F-Measure declines.The authors select tail size 20 to balance rejection with correct classification on ILSVRC 2012 validation images.
- Top Classes to be considered for revision: α = 10 provides optimal performance for revising top-ranked classes during FC8-score calibration.Lower α values reduce F-Measure, while values above 10 provide almost no additional gain.
- Distance Measures: Euclidean-cosine distance performs best among the tested distance measures, while Euclidean distance performs similarly.Cosine distance and Euclidean distance are marginally worse than euclidean-cosine distance in the reported experiments.
7 Qualitative Examples
Qualitative examples show that OpenMax can fail on visually similar unknown classes and can sometimes reduce the probability of a correct validation image enough to produce an error.
- Qualitative Examples: OpenMax fails on unknown objects visually similar to known ILSVRC 2012 classes, including close or distinct ILSVRC 2010 classes.The examples include both near-class variations and visually similar objects from distinct classes.
- Qualitative Examples: OpenMax can incorrectly reduce the probability of a correctly classified ILSVRC 2012 validation image.In the cited example, SoftMax produces the correct answer while OpenMax gets the example wrong.
8 Confusion Map of Mean Activation Vectors
OpenMax rejection depends on distances from the mean activation vectors of the highest-scoring FC8 classes. This makes fine-grained variants of known classes and nearby adversarial modifications less likely to be rejected.
- Confusion Map of Mean Activation Vectors: Unknown-class rejection uses distances to the mean activation vectors of the highest-scoring FC8 classes, rather than the closest mean activation vector overall.The distinction motivates considering confusion among existing classes.
- Confusion Map of Mean Activation Vectors: Fine-grained variants of known classes and adversarial images modified toward nearby classes are less likely to be rejected.Their proximity in the relevant activation representation limits OpenMax detection of these cases.
- Confusion Map of Mean Activation Vectors: Figure 6 compares fooling and open-set detection accuracy across uncertainty thresholds and EVT tail sizes for OpenMax and SoftMax.The figure also relates increased rejection to reduced rejection accuracy on true validation images.
9 Comparison with the 1-vs-set algorithm.
The paper compares its deep-network extension with a linear 1-vs-set open-set algorithm by training both approaches on FC8 data from the 1000 ImageNet classes.
- Comparison with the 1-vs-set algorithm: A linear SVM and a 1-vs-set machine are trained with liblinear on training samples from 1000 classes for comparison.The comparison tests whether applying another open-set model to FC8 data could outperform the paper’s direct deep-network extension.