PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenGAN: Open-Set Recognition via Open Data Generation

Shu Kong, Deva Ramanan

arXiv:2104.02939v3cs.CV

TL;DR

Open-set recognition must identify test examples outside the K closed-set classes, but existing outlier-based and GAN-based approaches have important limitations. OpenGAN combines adversarial fake open-data generation with feature-space discrimination and validation-based model selection, and it significantly outperforms prior methods across image recognition and semantic segmentation.

  • Problem

    Open-set recognition must discriminate test data outside K closed-set classes, while outlier-trained discriminators generalize poorly and GAN-based likelihood functions suffer from unstable training.

  • Method

    OpenGAN trains an open-vs-closed discriminator on OTS features, combines real open data with adversarially synthesized fake open data, and selects discriminators using validation.

  • Results

    OpenGAN significantly outperforms prior open-set methods across image recognition and semantic segmentation, achieving state-of-the-art open-set discrimination.

  • Takeaways & Limitations

    A carefully selected GAN discriminator can achieve state-of-the-art open-set discrimination, including when validation outliers are sparse or strongly biased.

  • Takeaways & Limitations

    Validation or training outlier sets may be biased, and GAN equilibrium can eventually make the discriminator incapable of separating closed-set from fake or open-set images.

Abstract

from arXiv · show

Real-world machine learning systems need to analyze test data that may differ from training data. In K-way classification, this is crisply formulated as open-set recognition, core to which is the ability to discriminate open-set data outside the K closed-set classes. Two conceptually elegant ideas for open-set discrimination are: 1) discriminatively learning an open-vs-closed binary discriminator by exploiting some outlier data as the open-set, and 2) unsupervised learning the closed-set data distribution with a GAN, using its discriminator as the open-set likelihood function. However, the former generalizes poorly to diverse open test data due to overfitting to the training outliers, which are unlikely to exhaustively span the open-world. The latter does not work well, presumably due to the instable training of GANs. Motivated by the above, we propose OpenGAN, which addresses the limitation of each approach by combining them with several technical insights. First, we show that a carefully selected GAN-discriminator on some real outlier data already achieves the state-of-the-art. Second, we augment the available set of real open training examples with adversarially synthesized "fake" data. Third and most importantly, we build the discriminator over the features computed by the closed-world K-way networks. This allows OpenGAN to be implemented via a lightweight discriminator head built on top of an existing K-way network. Extensive experiments show that OpenGAN significantly outperforms prior open-set methods.

1. Introduction

Open-set recognition addresses failures when test data falls outside the K closed-set classes, a safety concern for systems such as autonomous vehicles. OpenGAN combines outlier exposure, adversarially synthesized open data, feature-space training, and validation-based discriminator selection to improve open-set discrimination.

  • Motivation: Open-set recognition discriminates data outside the K closed-set classes, which standard K-way systems may misclassify as known categories.Such errors can be critical in autonomous vehicles when unfamiliar objects require different obstacle-avoidance plans.
  • Prior approaches: GAN discriminators offer an open-set likelihood function, but unstable GAN training has limited their effectiveness.This approach learns the closed-set distribution and uses the discriminator for open-set discrimination.
  • Prior approaches: Outlier Exposure trains a binary open-vs-closed discriminator, but biased training outliers may not span the open world and can cause poor generalization.The limitation is especially relevant when testing data differs from the available open-training examples.
  • OpenGAN: OpenGAN augments real open-training examples with adversarially synthesized fake open data to improve the discriminator’s coverage of open-set variation.The generator is trained to produce difficult examples that fool the classifier rather than merely easy synthetic negatives.
  • Results: OpenGAN achieves state-of-the-art open-set discrimination after selecting the discriminator with validation on real outlier examples, including under sparse or biased validation.The paper reports significant improvements over prior work for both image recognition and semantic segmentation.
  • OpenGAN: OpenGAN trains GANs over OTS features from closed-world K-way networks instead of RGB pixels, enabling a lightweight discriminator head on an existing network.The approach directly uses the discriminator as the open-set likelihood function rather than relying on reconstruction error.

4. Experiment

OpenGAN is evaluated across open-set discrimination, cross-dataset recognition, and semantic segmentation, with comparisons spanning feature- and pixel-based methods. Across these setups, OTS features and adversarially generated open data support stronger performance and generalization than several alternatives.

  • Experimental setups: Experiments cover single-dataset open-set discrimination, cross-dataset open-set recognition, and pixel-level open-set discrimination in semantic segmentation.Setup-I uses class-based open/closed splits, Setup-II uses cross-dataset open examples, and Setup-III evaluates semantic-segmentation pixels.
  • Implementation: OpenGANfea uses a lightweight discriminator and generator over penultimate-layer OTS features from existing K-way networks, while OpenGANpix processes pixels.The feature-based discriminator and generator are each 2MB; the pixel-based comparison uses a CNN of approximately 14MB.
  • Setup-II: Cross-dataset open-set recognition: OpenGANfea performs best in cross-dataset open-set recognition, while feature-based methods outperform pixel-based methods and OpenMax performs poorly in this setting.The comparison reports both AUROC and macro-averaged F1 over K+1 classes, with TinyImageNet as the closed set and four datasets as open sets.
  • Setup-II: Cross-dataset open-set recognition: Adversarially generated open data improves cross-dataset generalization over CLS, although OpenGAN can still perform poorly when training and testing open-set data differ substantially.OpenGAN-0, which does not train on open examples, performs as well as OpenGANfea in the reported diagnostic analysis.
  • Setup-III: Open-set semantic segmentation: For Cityscapes open-set segmentation, OpenGAN-0fea achieves 0.709 AUROC versus 0.549 for OpenGAN-0pix, while feature-based methods generally outperform pixel-based methods.Generated feature patches capture open-set objects outside the 19 closed-set classes, whereas pixel-based generated patches are visually realistic but perform worse.

5. Conclusion

OpenGAN combines feature-based discrimination with adversarially synthesized open data for open-set recognition, achieving state-of-the-art performance across image recognition and semantic segmentation.

  • OpenGAN trains an open-vs-closed classifier on OTS features rather than pixels.This is one of its two stated technical insights.
  • OpenGAN adversarially synthesizes fake open data to augment available open-training data.The generated examples supplement real open-training examples.
  • OpenGAN achieves state-of-the-art open-set discrimination after selecting its GAN discriminator with real outlier validation examples.The result remains effective when validation outliers are sparsely sampled or strongly biased.
  • OpenGAN significantly outperforms prior methods on both open-set image recognition and semantic segmentation.

Outline

The appendix outlines OpenGAN’s implementation details, segmentation setup, validation procedures, comparisons, visualizations, and failure cases or limitations.

  • Outline: The appendix expands OpenGAN’s techniques with architecture, model-selection, and training details.
  • Model architectures: It includes the architecture design for OpenGANfea.
  • Semantic segmentation setup: It details open-set semantic-segmentation data statistics and training batch construction.
  • Validation and tuning: It describes model selection and hyper-parameter tuning on validation sets.
  • Additional analyses: Additional sections cover generative statistical methods, quantitative comparisons, synthesized-image visuals, and open-set segmentation results.
  • Failure cases and limitations: A dedicated section reports failure cases and limitations.

6. Model Architecture

OpenGANfea uses compact MLPs over OTS features, while OpenGANpix uses convolutional CycleGAN-style networks for pixel data.

  • OpenGANfea: OpenGANfea uses MLP generator and discriminator networks because it operates on OTS features.OpenGANpix instead uses CNN architectures because it operates on pixels.
  • OpenGANfea: OpenGANfea is compact at approximately 2MB because its MLP operates on low-dimensional OTS features.The examples describe feature vectors such as 512 dimensions.
  • OpenGANfea: The OpenGANfea discriminator maps a D-dimensional feature through fully connected, normalization, and LeakyReLU layers to a sigmoid output.
  • OpenGANfea: The OpenGANfea generator maps a 64-dimensional random vector to a D-dimensional feature through MLP layers and a Tanh output.
  • Feature dimensions: For classification, D equals 512 from ResNet18; for segmentation, D equals 720 from HRNet’s penultimate layer.
  • OpenGANpix: OpenGANpix follows CycleGAN architectures with stride changes for image resolutions such as CIFAR 32x32 and TinyImageNet 64x64.
  • OpenGANpix: OpenGANpix has approximately 14MB and 11MB generator and discriminator models, respectively, to support high-dimensional RGB images.

7. Setup for Open-Set Semantic Segmentation

The Cityscapes setup treats historically ignored, non-noise pixels as open-set segmentation examples, using HRNet features and separate training, validation, and test splits.

  • Open-set data: Cityscapes historically ignored a sizeable portion of pixels outside its K closed-set classes, which are repurposed as open-set examples.This avoids introducing artificial open-set objects for the study.
  • Open-set data: The benchmark ignores 1010 “Other” pixels.
  • Data split: The training split uses 2,965 images, the validation split 10 images, and the test split 500 images.
  • Data statistics: Training data provide 334M closed-set pixels and 44M open-set pixels.
  • Data statistics: Validation data provide 1M closed-set pixels and 0.2M open-set pixels.
  • Data statistics: Test data provide 56M closed-set pixels and 2M open-set pixels.
  • Open-set data: Rectification-border, ego-vehicle, and out-of-roi pixels are excluded because they are known sensor- or viewpoint-related noise.
  • Feature extraction: HRNet penultimate-layer embeddings provide 720-dimensional per-pixel features before the 19-way classifier.Other layers did not produce significant performance differences.

8. Model Selection

Model selection is crucial because GAN training is unstable, and the discriminator—not the generator—must be selected for open-set recognition. Validation-selected models generalize to test sets, while longer training does not necessarily improve performance.

  • 8. Model Selection: GAN instability makes model selection crucial for OpenGAN open-set recognition.Unlike typical GAN use, OpenGAN selects discriminator checkpoints because the discriminator serves as the open-set likelihood function.
  • 8. Model Selection: Intermediate GAN checkpoints can achieve the best open-set discrimination performance.Open-set performance is evaluated across training epochs for OpenGAN-0fea and OpenGAN-0pix.
  • 8. Model Selection: Training fake-vs-real performance does not correlate with testing open-vs-closed performance.The scatter analysis compares checkpoint performance across three open-set recognition datasets.
  • 8. Model Selection: Validation-selected models generalize to test sets in the reported experiments.This generalization is demonstrated in Tables 3 and 4 of the main paper.

9. Hyper-Parameter Tuning

OpenGAN balances real open examples and generated fake open data through loss weights λo and λG. Validation performance reliably reflects test performance, although λG has no clear test-time correlation and depends on available real open data.

  • 9. Hyper-Parameter Tuning: OpenGAN training uses three loss terms for real closed pixels, real open pixels, and synthesized fake open pixels.The weights λo and λG balance the contributions of real open data and generated data.
  • 9. Hyper-Parameter Tuning: The real-open-data weight is fixed at λo = 1 because sparsely sampled open examples are presumed as important as closed examples.The generated-data weight λG is tuned separately.
  • 9. Hyper-Parameter Tuning: λG is varied from 0.05 to 0.90 to study how synthesized data affect training.The tested values are 0.05, 0.10, 0.15, 0.20, through 0.90.
  • 9. Hyper-Parameter Tuning: Lower λG is generally preferable when more real open-set training data are available.The experiments do not show a clear correlation between λG and test-time performance, which the authors attribute to random initialization.

10. Statistical Models for Open-Set

Open-set recognition can use statistical models over features extracted from closed-world K-way networks. The appendix examines GAN checkpoints, hyper-parameter effects, feature representations, and lightweight statistical alternatives for open-set likelihood estimation.

  • 10. Statistical Models for Open-Set: Statistical open-set pipelines extract and process K-way-network features, then estimate closed-set probability or distance for thresholding.Possible models include class centroids, nearest neighbors, Gaussian mixtures, and class-conditional GMMs.
  • 10. Statistical Models for Open-Set: OpenGAN-0 uses GAN discriminators as open-set likelihood functions, with intermediate checkpoints examined across training epochs.Figure 9 evaluates OpenGAN-0pix and OpenGAN-0fea on three standard open-set datasets.
  • 10. Statistical Models for Open-Set: Figure 10 compares fake-vs-real training performance with open-vs-closed testing performance across three datasets and epochs 0–50.The ideal relationship is a diagonal from bottom-left to top-right, but the caption reports no correlation.
  • 10. Statistical Models for Open-Set: Validation and test open-set performance align well when tuning λG, despite validation using 10 images and testing using 500.The plotted model is OpenGANfea-1000.
  • 10. Statistical Models for Open-Set: Intermediate-layer features are important because logits and softmax may be too invariant for open-set recognition.Figure 13 uses t-SNE plots to compare feature representations of open and closed testing data.
  • 10. Statistical Models for Open-Set: A street-shop example illustrates that open-set pixel recognition can identify a testing scene absent from comparable training examples.The authors manually searched the training set and found no other similar clothing-selling street-shops.
  • 10. Statistical Models for Open-Set: Spatial pooling and PCA reduce ResNet18 features from 512 dimensions to 50 without sacrificing performance.The reduction is important for learning second-order covariance statistics such as those used by GMMs.
  • 10. Statistical Models for Open-Set: Feature extraction and PCA reduce GMM covariance storage by 10^5× to 3.3×10^4 parameters and 128KB.The comparison is between pooled/PCA-reduced features and the original 512×7×7 activations.

11. Further Quantitative Results

Additional comparisons under Setup-I reinforce OpenGAN’s reported advantage over prior open-set methods. OpenGAN-0fea is identified as the best-performing variant in the listed comparison.

  • 11. Further Quantitative Results: OpenGAN outperforms the additional prior methods listed in the appendix under Setup-I.The comparison includes Entropy, GMM, CGDL, OpenHybrid, and RPL++.
  • 11. Further Quantitative Results: OpenGAN-0fea achieves the state-of-the-art in the Table 7 AUROC comparison.Table 7 measures open-set discrimination under Setup-I using AUROC.

12. Visualization of Generated Images

The appendix visualizes real and synthesized examples for OpenGAN variants, showing how feature-based and pixel-based generation differ in open-set segmentation.

  • OpenGAN-0pix generates visual “fake” images for intuitive inspection under Setup-I and Setup-II.The visualizations include closed-training, synthesized fake, closed-testing, and open-testing examples.
  • OpenGAN-0fea synthesizes Cityscapes patches analytically by matching generated features to nearest-neighbor training pixel features.The associated RGB patch is centered at the pixel whose feature is nearest in L1 distance.
  • 0.709 AUROC for OpenGAN-0fea versus 0.549 AUROC for OpenGAN-0pix on open-set segmentation.The feature-based method outperforms the pixel-based method despite the latter producing realistic color and tone.
  • OpenGAN-0fea’s synthesized patches capture open-set objects outside Cityscapes’ 19 defined classes.Examples include bridges, vehicle logos, and the backs of traffic signs.

13. Visual Results of Open-Set Segmentation

Visual results on Cityscapes show OpenGANfea capturing most open-set pixels and notably outperforming the other evaluated methods.

  • OpenGANfea captures most open-set pixels and notably outperforms the other methods in qualitative Cityscapes segmentation results.The testing set contains real open-set examples never seen during training.

14. Failure Cases and Limitations

The visualizations illustrate classification failures, while the limitations discussion highlights GAN equilibrium, validation-set bias, and untested adversarial robustness.

  • OpenGAN can fail when its discriminator assigns incorrect closed-set likelihoods, producing incorrect image or per-pixel predictions.The appendix points to confidence scores and thresholded predictions as examples of such failures.
  • GAN equilibrium can eventually make the discriminator incapable of distinguishing closed-set from fake or open-set images.Model selection on a validation set empirically improves performance, but does not remove this risk.
  • Validation-set bias may catastrophically hurt final open-set recognition performance, including when outlier training examples are available.The authors advise constructing reliable training and validation sets using prior knowledge.
  • OpenGAN’s robustness to adversarial input perturbations remains uninvestigated and is left for future work.
  • OpenGAN-0pix produces realistic-looking patches but reaches 0.549 AUROC, below OpenGAN-0fea’s 0.709 AUROC for open-set segmentation.OpenGAN-0fea’s synthesized patches capture several unknown objects outside Cityscapes’ 19 semantic classes.
Loading 2104.02939v3…