PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Set Recognition: a Good Closed-Set Classifier is All You Need?

Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman

arXiv:2110.06207v2cs.CVcs.LG

TL;DR

Open-set recognition asks whether a test sample belongs to a classifier’s known semantic classes, but the relationship between closed-set accuracy and open-set detection had not been fully established. The paper measures this relationship, strengthens a maximum-logit baseline, and introduces semantic-novelty benchmarks. It finds strong closed/open-set correlation and negligible performance difference between the improved baseline and existing state-of-the-art methods.

  • Problem

    OSR must identify semantic novelty beyond known classes, while existing benchmarks are small and lack clear definitions of semantic classes.

  • Method

    The paper studies closed/open-set performance across settings, improves a maximum-logit baseline using closed-set advances, and constructs the Semantic Shift Benchmark.

  • Results

    Closed-set and open-set performance are strongly correlated, and the improved MLS baseline is competitive with or outperforms state-of-the-art methods, with negligible difference from the improved state-of-the-art method.

  • Takeaways & Limitations

    A well-trained closed-set classifier using maximum logit scores can be sufficient to achieve competitive OSR performance, while SSB enables controlled study of semantic novelty.

  • Takeaways & Limitations

    OSR models remain far from perfect and cannot be exclusively relied upon in practical deployment, making representative training-data curation important.

Abstract

from arXiv · show

The ability to identify whether or not a test sample belongs to one of the semantic classes in a classifier's training set is critical to practical deployment of the model. This task is termed open-set recognition (OSR) and has received significant attention in recent years. In this paper, we first demonstrate that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We find that this relationship holds across loss objectives and architectures, and further demonstrate the trend both on the standard OSR benchmarks as well as on a large-scale ImageNet evaluation. Second, we use this correlation to boost the performance of a maximum logit score OSR 'baseline' by improving its closed-set accuracy, and with this strong baseline achieve state-of-the-art on a number of OSR benchmarks. Similarly, we boost the performance of the existing state-of-the-art method by improving its closed-set accuracy, but the resulting discrepancy with the strong baseline is marginal. Our third contribution is to present the 'Semantic Shift Benchmark' (SSB), which better respects the task of detecting semantic novelty, in contrast to other forms of distribution shift also considered in related sub-fields, such as out-of-distribution detection. On this new evaluation, we again demonstrate that there is negligible difference between the strong baseline and the existing state-of-the-art. Project Page: https://www.robots.ox.ac.uk/~vgg/research/osr/

1 INTRODUCTION

The paper argues that open-set recognition performance is strongly tied to closed-set classifier accuracy, then leverages this relationship to strengthen a simple baseline and proposes benchmarks focused on semantic novelty.

  • 1 INTRODUCTION: OSR requires classifiers to recognize trained categories while identifying images from previously unseen classes.This extends the closed-set recognition setting, where training and test categories remain the same.
  • 1 INTRODUCTION: Closed-set and open-set performance are highly correlated across datasets, objectives, architectures, and an ImageNet-scale evaluation.The finding challenges the expectation that stronger closed-set classifiers necessarily overfit and perform poorly at OSR.
  • 1 INTRODUCTION: Improving closed-set accuracy and using maximum logit scores can make an OSR baseline competitive with or stronger than state-of-the-art methods.The paper applies stronger augmentation, learning-rate schedules, label smoothing, and MLS-based open-set scoring.
  • 1 INTRODUCTION: The improved approach achieves new state-of-the-art figures on four of six OSR benchmarks, while the improved existing state-of-the-art method differs negligibly from the MLS baseline.These results enable a closer assessment of reported progress among more complex OSR methods.
  • 1 INTRODUCTION: The Semantic Shift Benchmark uses larger-scale datasets and explicit semantic-class definitions to evaluate semantic novelty more precisely.Its open-set splits target semantic novelty and distinguish OSR from low-level distribution shift and related tasks.

2 RELATED WORK

The related-work discussion positions OSR among deep-learning methods and neighboring novelty-detection fields, while emphasizing that OSR specifically concerns semantic novelty.

  • 2 RELATED WORK: OSR research includes OpenMax, GAN-based approaches, and other methods designed to detect samples outside known training classes.OpenMax uses Extreme Value Theory, while OSRCI generates non-known-class images for training.
  • 2 RELATED WORK: The paper compares against ARPL and OpenHybrid, arguing that a simple baseline can match or outperform these more complex state-of-the-art methods.The comparison is framed in the controlled OSR setting without extra data for training or model selection.
  • 2 RELATED WORK: OSR is related to out-of-distribution, novelty, anomaly, and novel-category discovery tasks, but specifically targets semantic novelty rather than every distributional shift.The proposed benchmarks are intended to preserve this distinction from neighboring fields.

3 CORRELATION BETWEEN CLOSED-SET AND OPEN-SET PERFORMANCE

The paper finds a strong positive relationship between closed-set accuracy and open-set recognition performance across standard benchmarks and ImageNet, including within architectures. It evaluates OSR using formal known/unknown decisions, standard methods, and large-scale semantic splits.

  • OSR formulation: OSR evaluates images from known and unseen classes by combining known-class prediction with a score indicating whether each image belongs to any known class.The experiments use AUROC for threshold-free evaluation of binary known/unknown decisions.
  • Standard benchmarks: The standard-benchmark experiments compare the MSP baseline with two ARPL variants across datasets containing training-known and evaluation-unseen classes.Models are trained on subsets of classes, with other classes reserved as unseen for evaluation.
  • Standard benchmarks: Closed-set and open-set performance are strongly correlated across standard OSR benchmarks, challenging the expectation that stronger classifiers overfit known classes and perform poorly on novelty detection.The standard-benchmark evaluation reports Pearson correlation ρ = 0.95 between Top-1 accuracy and AUROC.
  • Interpretation: The authors connect the observed trend to calibration: lower generalization error under proper scoring rules is expected to correspond to better calibration and more informative confidence.The calibration argument is presented theoretically under infinite-data and generalization assumptions.
  • ImageNet evaluation: ImageNet experiments also show positive closed/open-set correlations, with ρ = 0.88 for Hard and ρ = 0.63 for Easy semantic splits across architectures.Within the ResNet family, the relationship strengthens to ρ = 1.00 for Easy and ρ = 0.99 for Hard.
  • ImageNet evaluation: A ViT model departs from the observed OSR trend, but it was pretrained on categories from the unseen splits before closed-set fine-tuning.The authors identify this pretraining overlap as a relevant qualification when interpreting the architecture comparison.

4 A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED?

The paper improves the OSR baseline by strengthening closed-set training and replacing softmax scoring with maximum logit scoring. These changes make the baseline competitive with or better than more complex methods across standard benchmarks, while improved state-of-the-art methods show nearly identical average performance.

  • Improved closed-set training makes the MSP baseline competitive with or stronger than state-of-the-art open-set models, achieving new state-of-the-art results on four of six benchmarks.The strategies include longer training, better augmentations, and label smoothing.
  • Maximum logit scoring substantially improves reported baseline performance, increasing average AUROC by 15.6% across the datasets.MLS uses logits rather than normalized softmax outputs for open-set scoring.
  • MLS surpasses existing state-of-the-art performance on SVHN, CIFAR+10, CIFAR+50, and TinyImageNet, averaging 0.7% better across the entire suite.
  • Augmenting OSRCI and ARPL+CS with improved training strategies boosts open-set performance in almost all cases.The augmented ARPL+CS achieves new state-of-the-art results on CIFAR+10 and CIFAR+50 and improves TinyImageNet by 4.3%.
  • The augmented ARPL+CS is almost indistinguishable from the improved MLS baseline, differing by only 0.03% in average open-set performance.
  • The findings suggest that many reported OSR gains may come from improved closed-set accuracy, which is often unreported for sophisticated methods.When ARPL and the baseline have comparable closed-set accuracy, their open-set performance differs negligibly.

5 SEMANTIC SHIFT BENCHMARK

The Semantic Shift Benchmark addresses small-scale OSR benchmarks and their unclear definitions of semantic classes by using ImageNet-scale and fine-grained evaluations. Its semantic difficulty splits reveal stronger performance differences than openness, while MLS remains comparable to the state-of-the-art method.

  • Existing OSR benchmarks are small-scale and lack a clear definition of semantic class, making it difficult to distinguish semantic novelty from related distribution-shift problems.
  • The proposed SSB uses ImageNet and fine-grained datasets with explicit semantic-novelty splits, including CUB, Stanford Cars, and FGVC-Aircraft.
  • ImageNet: ImageNet open-set splits are formed from ImageNet-21K-P using semantic distances in the hierarchy relative to the ImageNet-1K closed-set classes.The authors select Easy and Hard splits from categories with different total semantic distances to the closed set.
  • Fine-grained datasets: Fine-grained splits use labelled visual attributes to group open-set classes as Easy, Medium, or Hard according to their similarity to training classes.CUB represents each class with attribute frequencies in a class-by-attribute matrix.
  • Evaluation protocol: SSB evaluates known-versus-unknown decisions with AUROC, closed-set accuracy, and OSCR on Easy and Hard splits.
  • Results: On SSB, strong closed-set classifiers produce good AUROC performance and MLS performs comparably to the state-of-the-art method.
  • Results: On ImageNet, the harder semantic split reduces AUROC by 5–6% for both methods, unlike openness, which depends only on the ratio of closed- to open-set classes.

6 CONCLUSION

The paper finds strong correlation between closed-set and open-set performance, showing that a well-trained classifier using maximum logit scoring can match or exceed sophisticated OSR methods. It also introduces the Semantic Shift Benchmark for controlled study of semantic novelty.

  • Closed-set and open-set performance are strongly correlated in open-set recognition.
  • A well-trained closed-set classifier using maximum logit scoring can compete with or outperform existing state-of-the-art OSR methods.
  • The paper proposes the Semantic Shift Benchmark to isolate semantic shift from other low-level distributional shifts.
  • The benchmark supports controlled study of semantic novelty, including stratification by the degree of semantic shift.

ETHICS STATEMENT

OSR is relevant to safe deployment because unknown categories may trigger unsafe forced classifications. The paper also cautions that OSR models are imperfect, datasets may contain personal data, and benchmark trends can have important limitations.

  • Forcing unknown objects into training categories in autonomous driving could lead to unsafe decisions.
  • OSR models are far from perfect and cannot be exclusively relied upon in practical deployment.
  • Training data should be carefully curated and representative of the target task rather than treated as automatically adequate by OSR.
  • CIFAR, TinyImageNet, and ImageNet may contain personal data for which consent was likely not obtained.
  • The figure’s correlation trend is less clear at very high accuracies, possibly because AUROC is high or the relationship weakens near saturation.

B ANALYSING THE CLOSED-SET AND OPEN-SET CORRELATION

The analysis explains why stronger cross-entropy classifiers can improve OSR: training increases known-class feature and weight magnitudes while uncertain samples retain lower norms. Maximum logit scoring preserves this magnitude signal, unlike softmax normalization.

  • The study trains VGG32 on CIFAR10 with feature dimensions D = 128 and D = 2 to analyze the closed/open-set relationship.
  • During training, known-class feature magnitudes increase substantially more than unknown-class feature magnitudes.
  • Cross-entropy encourages larger correct-class weights and feature norms because both reduce the loss.
  • For hard or uncertain examples, the loss reduces competing class logits, encouraging lower feature norms across both D = 2 and D = 128 settings.
  • Stronger cross-entropy models project features farther from the origin while preserving lower norms for uncertain samples, supporting OSR performance.
  • Maximum logit scoring retains feature-magnitude information that softmax normalization cancels, providing the proposed OSR score.

C IMPROVING OPEN-SET PERFORMANCE WITH STRONGER CLOSED-SET

The paper improves OSR by strengthening closed-set training and using maximum logit scoring. On TinyImageNet, these standard recognition improvements raise AUROC substantially, with ensembles providing a further gain.

  • The experimental breakdown evaluates improved baseline training and scoring strategies on five TinyImageNet known/unknown splits.
  • Maximum logit scoring gives better open-set AUROC, while softmax scoring gives better OSCR in the baseline comparison.
  • 69.6% AUROC is achieved with maximum logit scoring after 100 training epochs, versus the literature’s 57.7% AUROC softmax-thresholding baseline.
  • Longer training and cosine annealing significantly enhance both closed-set and open-set performance.
  • 83.0% AUROC results from combining longer training, improved augmentations, warmup, and label smoothing.
  • 84.0% AUROC is reached by bootstrapping the training data and training K = 5 ensembles.
  • The improved training approach is applied to prior OSR methods, including OSRCI’s multi-stage GAN-based classifier training.

D.3 PROPOSED BENCHMARKS

The proposed benchmark experiments use pretrained or newly trained models to compare MLS with ARPL+ across ImageNet and fine-grained datasets. ARPL+CS was computationally infeasible on the proposed datasets, while ARPL+ training remained feasible.

  • ImageNet: ImageNet evaluation applies MLS directly to a cross-entropy-pretrained ResNet50 and fine-tunes the same model for ARPL+.ARPL+ is fine-tuned for 10 epochs using its optimization strategy.
  • FGVC datasets: FGVC experiments train MLS and ARPL+ for 600 epochs with cosine annealing, warmup, retuned augmentation, and label smoothing.They use a ResNet50 backbone with 448 × 448 images, initialized from pretrained weights.
  • Computational cost: ARPL+CS is computationally infeasible on the proposed datasets because its memory demands restrict batch sizes and greatly extend training.On FGVC datasets, training with larger hardware took a week and performed slightly below ARPL+.

E COMPARISONS WITH OTHER DEEP LEARNING BASED OSR METHODS

The improved MLS baseline is compared with deep-learning OSR methods and related OoD evaluations. Strong closed-set models substantially improve baseline performance, with reported gaps to state-of-the-art becoming negligible.

  • OSR comparisons: The improved baseline significantly outperforms reported baseline figures and surpasses state-of-the-art OSR models on several standard benchmarks.The comparison table reports AUROC percentages across methods and backbone architectures.
  • OSR comparisons: Training OSRCI and ARPL+CS on top of the improved baseline can further boost their OSR performance.
  • OSR comparisons: The discrepancy between the improved baseline and state-of-the-art OSR methods is negligible.
  • OoD comparisons: Four CIFAR100-trained ResNet models show closed-set and OoD performance correlation of ρ = 0.97 when CIFAR10 is used as OoD.This trend resembles the ImageNet OSR evaluation.
  • OoD comparisons: The OoD experiments compare Gaussian Noise, SVHN, and CIFAR10 as OoD datasets relative to CIFAR100.

F.3 EVALUATION ON OOD BENCHMARKS

The full OoD evaluation tests MLS on standard datasets and connects performance to the paper’s analysis of semantic novelty and categorization. Strong closed-set classifiers make MLS highly competitive, especially in near-OoD settings.

  • OoD benchmark results: MLS applied to strong WideResNet-40 models substantially closes the gap between the MSP baseline and state-of-the-art OoD methods.The benchmark suite includes Places365, Textures, LSUN-Crop, LSUN-Resize, iSUN, and SVHN.
  • OoD benchmark results: The MLS baseline can match or exceed recent OoD state-of-the-art methods, including Virtual Outlier Synthesis, in some cases.
  • OoD benchmark results: MLS performs best in near-OoD settings, while CIFAR100-trained models detect Gaussian Noise less effectively than CIFAR10 images as OoD.The finding suggests that OoD and OSR may involve different, possibly orthogonal, solutions.
  • Categorization: Reliable OSR requires modeling within-category invariances and category-sensitive axes of variation.These axes can include pose, lighting, gender, background, or more abstract features.
  • Categorization: Current OSR benchmarks are ill-posed because diverse axes, including semantically meaningless low-level features, can distinguish their classes.Fine-grained datasets constrain the relevant semantic variation more strongly.
  • Split construction: The proposed splits are constructed from dataset-specific class attributes, hierarchies, or name similarities to control semantic difficulty.CUB uses attribute-based similarity; Stanford Cars and FGVC-Aircraft use class structures and attributes.

H.2 SEMANTIC SHIFT BENCHMARK EXAMPLES

The Semantic Shift Benchmark illustrates open-set classes at multiple difficulty levels across fine-grained datasets and ImageNet. Hard classes are selected to be more visually similar to corresponding closed-set classes, and evaluation reports AP for known-versus-unknown decisions.

  • Fine-grained examples: The SSB examples combine Medium and Hard into a single Hard split for evaluation.Figures 9–11 show Easy, Medium, and Hard examples for CUB, Stanford Cars, and FGVC-Aircraft.
  • ImageNet examples: ImageNet examples show Easy and Hard open-set classes alongside their most similar closed-set classes.
  • Difficulty structure: Hard open-set classes are far more visually similar to their corresponding closed-set classes than Easy open-set classes.
  • Benchmark availability: The Semantic Shift Benchmark is publicly available through the project page.
  • Evaluation: Average precision measures the binary known-versus-unknown decision for the proposed benchmark’s Easy, Medium, and Hard splits.AP is used because it is suited to class imbalance at test time.
Loading 2110.06207v2…