PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Models for Open Set Recognition

Hongjie Zhang, Ang Li, Jie Guo, Yanwen Guo

arXiv:2003.12506v2cs.CV

TL;DR

Open set recognition must identify unknown classes absent from training, but existing representations focus on known classes and can mishandle outliers. OpenHybrid jointly learns a shared embedding for classification and flow-based density estimation, achieving state-of-the-art benchmark performance and addressing a common flow-model likelihood issue in the reported experiments.

  • Problem

    Existing open set methods use discriminative embedding spaces focused on known classes, while flow-based models may assign higher likelihood to outliers.

  • Method

    OpenHybrid jointly trains an encoder, an inlier classifier, and a flow-based density estimator in a shared latent space, using density thresholds to identify unknown samples.

  • Results

    OpenHybrid achieves state-of-the-art open set recognition performance, outperforming non-flow-based baselines across six datasets and clearly separating known and unknown likelihood distributions.

  • Takeaways & Limitations

    Joint representation learning is a key contributor to OpenHybrid's performance and empirically resolves the reported higher-likelihood-to-outliers issue across the evaluated datasets.

  • Takeaways & Limitations

    The reported evaluation includes a specific CIFAR10-known versus SVHN-unknown setting for examining flow-model likelihood behavior.

Abstract

from arXiv · show

Open set recognition requires a classifier to detect samples not belonging to any of the classes in its training set. Existing methods fit a probability distribution to the training samples on their embedding space and detect outliers according to this distribution. The embedding space is often obtained from a discriminative classifier. However, such discriminative representation focuses only on known classes, which may not be critical for distinguishing the unknown classes. We argue that the representation space should be jointly learned from the inlier classifier and the density estimator (served as an outlier detector). We propose the OpenHybrid framework, which is composed of an encoder to encode the input data into a joint embedding space, a classifier to classify samples to inlier classes, and a flow-based density estimator to detect whether a sample belongs to the unknown category. A typical problem of existing flow-based models is that they may assign a higher likelihood to outliers. However, we empirically observe that such an issue does not occur in our experiments when learning a joint representation for discriminative and generative components. Experiments on standard open set benchmarks also reveal that an end-to-end trained OpenHybrid model significantly outperforms state-of-the-art methods and flow-based baselines.

1 Introduction

Open set recognition addresses the unrealistic closed-set assumption by requiring models to classify known samples while rejecting unknowns. OpenHybrid jointly learns a representation for classification and flow-based density estimation, outperforming prior methods across benchmarks.

  • Problem: Closed-set classifiers force out-of-distribution samples into known classes, whereas open-set recognition adds an unknown category.The open-set test set contains both known and unknown classes.
  • Existing approaches: Existing methods estimate training-sample distributions in discriminative embedding spaces and threshold probability scores to detect unknowns.SoftMax thresholding has been reported to achieve only sub-optimal open-set recognition solutions.
  • OpenHybrid: Joint embedding learning empirically avoids the higher likelihood for OOD inputs reported as a common flow-based-model failure.The authors suggest the learned deep representation may better accommodate multimodal input distributions.
  • OpenHybrid: OpenHybrid learns a joint representation shared by a classifier and a flow-based density estimator, trained end-to-end.The framework combines discriminative and generative components while preserving separate classification and density-estimation branches.

2 Related Work

Related work spans discriminative and generative open-set methods, OOD detection, and flow-based density models. OpenHybrid builds on hybrid modeling but differs by sharing a visual representation between classification and density estimation.

  • Discriminative methods: Discriminative approaches include traditional classifiers and deep methods such as OpenMax, which calibrates SoftMax outputs using Weibull modeling.Earlier traditional methods often require feature engineering and may not scale well.
  • Generative methods: Generative approaches use GANs to synthesize unknown samples and improve decision boundaries between known and unknown classes.Generative OpenMax uses a conditional GAN to create mixtures of known classes for classifier fine-tuning.
  • Open-set recognition: Open-set recognition extends related OOD, outlier, and novelty detection settings by requiring discriminative classification among known classes.OOD detectors alone do not need discriminative power within known classes.
  • Flow-based methods: Flow-based models provide invertible density estimators, but practical limitations include weak per-layer nonlinear transformations and biased infinite-series truncation in I-ResNet.These models estimate likelihoods through tractable transformations and Jacobian determinants.
  • Flow-based OOD detection: Flow-based OOD detectors can assign higher likelihood to OOD inputs, motivating techniques such as Outlier Exposure, mixture priors, and typicality tests.OpenHybrid follows the hybrid direction while sharing a common visual representation for its classifier and flow model.

3 Our Approach

OpenHybrid jointly learns a latent representation for known-class classification and flow-based density estimation. Its encoder, classifier, and density estimator are trained together, then density thresholds identify unknown samples while classifier scores assign known-class labels.

  • OpenHybrid consists of an encoder F for latent representations, a classifier C for known classes, and a flow-based module D for density estimation.
  • The model learns a classifier and flow density estimator from a joint representation rather than relying only on discriminative features.The classifier is connected directly to the encoder output, avoiding dependence on the invertible flow transformation.
  • The classification component trains encoder F and classifier C with cross-entropy loss over labeled known classes.The class probability p(yj|xi; Θf, Θc) is produced from the classifier applied to the encoder representation.
  • The density component estimates the distribution of learned latent representations F(xi) using a flow-based model and maximum likelihood.Flow transformations enable tractable exact density evaluation through the change-of-variables formula.
  • The complete training objective combines classification loss and density-estimation loss as L({Θf, Θc, Θd}) = LC({Θf, Θc}) + λLD({Θf, Θd}).The scaling factor λ controls the density-loss contribution and is set to 1 in the reported experiments.
  • At inference, samples with density below threshold τ are assigned to the unknown class k + 1; otherwise, the classifier assigns the highest-scoring known class.The threshold is estimated from training samples, and the reported procedure uses samples without data augmentation.

4 Experiments

OpenHybrid is evaluated across standard open-set and cross-dataset benchmarks against non-flow-based and flow-based methods. Results show stronger performance from joint training, especially on complex natural-image datasets, while visualizations illustrate improved density separation and latent organization.

  • Comparison with Non-flow-based Methods: OpenHybrid outperforms non-flow-based baselines across six datasets, with gains of 6% on CIFAR and 5% on TinyImageNet over the second-best method.MNIST shows only minor improvement because results are nearly saturated, whereas gains are larger on complex natural-image datasets.
  • Comparison with Flow-based Methods: OpenHybrid consistently outperforms flow-based baselines under all open-set benchmarks using AUROC.The comparison uses a unified threshold for flow-based methods and follows other methods’ protocols for fair comparisons.
  • Comparison with Flow-based Methods: The F-score reaches 0.865 on CIFAR10, compared with 0.673 for DIGLM and 0.701 for DIGLM+OE.F-score is used in some experiments because it also measures in-distribution classification performance.
  • Cross-dataset OOD settings: Cross-dataset evaluation remains competitive, with higher gains of +10% when training on CIFAR100 and testing on CIFAR10.The latter direction is described as harder, probably because it uses more training classes.
  • The benefit of joint training: Jointly training the encoder, classifier, and flow-based model consistently outperforms using a fixed pretrained encoder.The comparison uses F-score with slack parameter s set to 80 for all datasets.
  • A study on the parameters: On MNIST, AUROC varies from 0.993 to 0.998 as λ increases from 0.5 to 2, suggesting limited sensitivity to this parameter.The authors note that λ can still be tuned to improve performance.
  • A visualization of the estimated density: End-to-end OpenHybrid separates unknown MNIST likelihoods from known samples, whereas DIGLM distributions nearly overlap and pretrained OpenHybrid remains less separated.The corresponding latent-space visualization also separates unknown digits from known classes and separates digits 6–9 unsupervised.
  • A disappeared issue of flow-based models: On CIFAR10-versus-SVHN detection, OpenHybrid achieves 0.998 AUROC and places nearly all unknown samples left of known samples in the likelihood histogram.DIGLM assigns higher likelihoods to unknown SVHN samples, while OE only partially addresses the overlap.

5 Conclusion

The paper presents OpenHybrid, combining a flow-based density estimator and discriminative classifier in a shared latent space for open set recognition. Experiments report state-of-the-art performance, with joint feature-space training eliminating a common flow-model issue and contributing to stronger recognition.

  • OpenHybrid combines a flow-based density estimator and discriminative classifier through a shared latent space.
  • Extensive experiments show that OpenHybrid achieves state-of-the-art open set recognition performance.
  • Joint feature-space learning empirically removes the tendency of flow-based models to assign larger likelihoods to out-of-distribution samples.
  • Ablation results identify joint training as an important contributor to OpenHybrid’s superior open set recognition performance.
Loading 2003.12506v2…